Kubernetes container connection to RDS instance in

Question

I have a Kubernetes cluster running in Amazon EC2 inside its own VPC, and I'm trying to get Dockerized services to connect to an RDS database (which is in a different VPC). I've figured out the peering and routing table entries so I can do this from the minion machines:

ubuntu@minion1:~$ psql -h <rds-instance-name>
Password:

So that's all working. The problem is that when I try to make that connection from inside a Kubernetes-managed container, I get a timeout:

ubuntu@pod-1234:~$ psql -h <rds-instance-name>
…

To get the minion to connect, I configured a peering connection, set up the routing tables from the Kubernetes VPC so that 10.0.0.0/16 (the CIDR for the RDS VPC) maps to the peering connection, and updated the RDS instance's security group to allow traffic to port 5432 from the address range 172.20.0.0/16 (the CIDR for the Kubernetes VPC).

Answer 1

With the help of Kelsey Hightower, I solved the problem. It turns out it was a Docker routing issue. I've written up the details in a blog post, but the bottom line is to alter the minions' routing table like so:

$ sudo iptables -t nat -I POSTROUTING -d <RDS-IP-ADDRESS>/32 -o eth0 -j MASQUERADE

Answer 2

Did you modify the source/destination checks as well?

Since your instance will be sending and receiving traffic for IPs other than the one assigned by your subnet, you'll need to disable source/destination checks.

See the image: https://coreos.com/assets/images/media/aws-src-dst-check.png