Docker Swarm - Network Overlay not connecting all

Question

I currently have 3 EC2 instances setup with Docker Swarm and Consul. I have 3 simple node apps spread across all 3 instances, and then nginx for routing on my swarm master.

Using the guide for Overlay Networks, I've created a new overlay called mynetand have every container connected to this network. Using docker inspect I can confirm that each node container and the nginx container all are connected to mynet and have an IP. However, after sshing in to my nginx container, it's only able to communicate with the node app that sits on the same host through the network, and not the ones on the other hosts in the cluster.

After running docker network inspect mynet, only two of my containers are listed (the nginx and one node instance that were on the same host), while the other two aren't listed, even though they have the name and IPs for that overlay network from docker inspect.

The idea is using an nginx reverse proxy to add any new containers that come online to it's config, but am stuck on getting containers to all connect. According to the Docker networking docs for overlay this should possible.

What am I doing wrong? Thanks.

EDIT:

After spamming docker network inspect mynet a few times, it looks like containers are constantly connecting and disconnecting from the network, even though I have all firewalls off and ports opened for testing purposes. What would be causing something like that?

Answer 1

Turns out there were two issues that's caused my problem.

One is I was using Kernel 3.13 which docker network overlay doesn't support. Upgrading to 3.16 fixed the issue. Second is there's a bug in Swarm 1.0.1 that does aggregate the network information correctly. It's been fixed here but as of time of writing it has not been released yet.