Docker Bridge Conflicts with Host Network

Question

Docker seems to be creating a bridge after a container starts running that then conflicts with my host network. This is not the default bridge docker0, but rather another bridge that is created after a container has started. I am able to configure the default bridge according to the older user guide link https://docs.docker.com/v17.09/engine/userguide/networking/default_network/custom-docker0/, however, I do not know how to configure this other bridge so it does not conflict with 172.17.

This current issue is then that my container cannot access other systems on the host network when this bridge becomes active.

Any ideas?

Version of docker:

Version 18.03.1-ce-mac65 (24312)

This is the bridge that gets created. Sometimes it is not 172.17, but sometimes it is.

br-f7b50f41d024 Link encap:Ethernet  HWaddr 02:42:7D:1B:05:A3  
      inet addr:172.17.0.1  Bcast:172.17.255.255  Mask:255.255.0.0

Answer 1

The bridge was created from docker-compose, which can be configured within the compose file.

Answer found here: Docker create two bridges that corrupts my internet access

Answer 2

When docker networks are created (e.g. using docker network create or indirectly through docker-compose) without explicitly specifying a subnet range, dockerd allocates a new /16 network, starting from 172.N.0.0/16, where N is a number that is incremented (e.g. N=17, N=18, N=19, N=20, ...). A given N is skipped if there is already a docker network in the range: either a custom docker network, or the default docker bridge.

When creating a new bridge network, you could specify explicitly the IP range to use to avoid the conflict. Unfortunately, doing so would require modifying every docker-compose.yaml file you encounter -- to explicitly provide a safe ip block. Generally, one would want to avoid including host-specific things inside compose files.

Instead, you can play with the networks that docker considers allocated. I'm outlining 3 methods below, which should allow you to force dockerd to "skip" subnets.

Method #1 -- create a dummy placeholder network

You can prevent the entire 172.17.0.0/16 from being used by dockerd (in future bridge networks) by creating a very small docker network anywhere inside 172.17.0.0/16.

Find 4 consecutive IPs in 172.17.* that you know are not in use in your host network, and sacrifice them in a "tombstone" docker bridge. Below, I'm assuming the ips 172.17.253.0, 172.17.253.1, 172.17.253.2, 172.17.253.3 (i.e. 172.17.253.0/30) are unused in your host network.

docker network create --driver=bridge --subnet 172.17.253.0/30 tombstone
# created: c48327b0443dc67d1b727da3385e433fdfd8710ce1cc3afd44ed820d3ae009f5

Note the /30 suffix here, which is 4 different IPs. In theory, the smallest valid network subnet should be a /31 which consists of a total of 2 IPs (network identifier + broadcast). Docker asks for a /30 minimum, probably to account for a gateway host, and another container. I picked .253.0 arbitrarily, you should pick something that's not in use in your environment. Also note that the identifier tombstone is nothing special, you can rename it to anything that will help you remember why it's there when you find it again several months later.

If you look at your routing table, you will see the entry for the new bridge:

# output of route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.5.1     0.0.0.0         UG    0      0        0 eth1
172.17.253.0    0.0.0.0         255.255.255.252 U     0      0        0 br-c48327b0443d
172.20.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
192.168.5.0     0.0.0.0         255.255.255.0   U     0      0        0 eth1

Note: My docker bridge is on 172.20.0.1 in my setup. I've modified bip in /etc/docker/daemon.json to do that. See this page for more details.

Now, if we create new docker networks, we can see that the rest of 172.17.0.0/16 is skipped, because the range is not entirely available.

docker network create foo_test
# c9e1b01f70032b1eff08e48bac1d5e2039fdc009635bfe8ef1fd4ca60a6af143
docker network create bar_test
# 7ad5611bfa07bda462740c1dd00c5007a934b7fc77414b529d0ec2613924cc57

The resulting routing table:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.5.1     0.0.0.0         UG    0      0        0 eth1
172.17.253.0    0.0.0.0         255.255.255.252 U     0      0        0 br-c48327b0443d
172.18.0.0      0.0.0.0         255.255.0.0     U     0      0        0 br-c9e1b01f7003
172.19.0.0      0.0.0.0         255.255.0.0     U     0      0        0 br-7ad5611bfa07
172.20.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
192.168.5.0     0.0.0.0         255.255.255.0   U     0      0        0 eth1

Notice that the rest of the IPs in 172.17.0.0/16 have not been used. The new networks reserved .18. and .19.. Sending traffic to any of your conflicting IPs outside that tombstone network would go via your host network.

You would have to keep that tombstone network around in docker, but not use it in your containers. It's a dummy placeholder network.

Method #2 -- bring down the conflicting bridge network

If you wish to temporarily avoid the IP conflict, you can bring the conflicting docker bridge down using ip link: ip link set dev br-xxxxxxx down. This will have the effect of removing the corresponding bridge routing entry in the routing table, without modifying any of the docker metadata.

This is arguably not as good as the method above, because you'd have to bring down the interface possibly every time dockerd starts, and it would interfere with your container networking if there was any container using that bridge.

If method 1 stops working in the future (e.g. because docker tries to be smarter and reuse unused parts of an ip block), you could combine both approaches: e.g. create a large tombstone network with the entire /16, not use it in any container, and then bring its corresponding br-x device down.

Method #3 -- reconfigure your docker bridge to occupy a subportion of the conflicting /16

As a slight variation of the above, you could make the default docker bridge overlap with a region of 172.17.*.* that is not used in your host network. You can change the default docker bridge subnet by changing the bridge ip (i.e. bip key) in /etc/docker/daemon.json (See this page for more details). Just make it a subregion of your /16, e.g. in a /24 or smaller.

I've not tested this, but I presume any new docker network would skip the remainder of 172.17.0.0/16 and allocate an entirely different /16 for each new bridge.

For the future:

I was looking for a way to configure dockerd with a different starting IP range for new network bridges, but couldn't find anything (as of version 1.13.1). Hopefully that changes in the future. Being able to specify ranges of IPs to exclude seems to be a valid thing to want.