DNS not working within docker containers when host

Question

The symptom is: the host machine has proper network access, but programs running within containers can't resolve DNS names (which may appear to be "can't access the network" before investigating more).

$ sudo docker run -ti mmoy/ubuntu-netutils /bin/bash
root@082bd4ead733:/# ping www.example.com
... nothing happens (timeout) ... ^C
root@082bd4ead733:/# host www.example.com
... nothing happens (timeout) ... ^C

(The docker image mmoy/ubuntu-netutils is a simple image based on Ubuntu with ping and host included, convenient here since the network is broken and we can't apt install these tools)

The issue comes from the fact that docker automatically configured Google's public DNS as DNS server within the container:

root@082bd4ead733:/# cat /etc/resolv.conf 
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN

nameserver 8.8.8.8
nameserver 8.8.4.4

This just works in many configurations, but obviously doesn't when the host runs on a network where Google's public DNS are filtered by some firewall rules.

The reason this happened is:

• Docker first tries configuring the same DNS server(s) on the host and within the container.
• The host runs dnsmasq, a DNS caching service. dnsmasq acts as a proxy for DNS requests, hence the apparent DNS server in the host's /etc/resolve.conf is nameserver 127.0.1.1, i.e. localhost.
• The host's dnsmasq listens only for requests comming from localhost and blocks requests coming from the docker container.
• Since using 127.0.1.1 within docker doesn't work, docker falls back to Google's public DNS, which do not work either.

There may be several reasons why DNS is broken within docker containers. This question (and answers) covers the case where:

• dnsmasq is used. To check whether this is the case:
  • Run ps -e | grep dnsmasq on the host. If the output is empty, you're not running dnsmasq.
  • Check the host's resolv.conf, it probably contains an entry like nameserver 127.0.1.1. If it contains nameserver 127.0.0.53, you're probably running systemd-resolved instead of dnsmasq. If so, you won't be able to use the solution forwading DNS requests to dnsmasq (the one using listen-address=172.17.0.1). systemd-resolved hardcodes the fact that it listens only on the 'lo' interface hence there's no easy way to adapt this solution. Other answers below will work with systemd-resolved.
• Google's public DNS is filtered. Run host www.example.com 8.8.8.8. If it fails or times out, then you are in this situation.

What are the solutions to get a proper DNS configuration in this configuration?

Answer 1

Since the automatic DNS discovery is guilty here, you may override the default setting in docker's configuration.

First, get the IP of the DNS server dnsmasq is using with e.g.:

$ sudo kill -USR1 `pidof dnsmasq`
$ sudo tail /var/log/syslog 
[...]
Apr 24 13:20:19 host dnsmasq[2537]: server xx.yy.zz.tt1#53: queries sent 0, retried or failed 0
Apr 24 13:20:19 host dnsmasq[2537]: server xx.yy.zz.tt2#53: queries sent 0, retried or failed 0

The IP addresses correspond to the xx.yy.zz.tt placeholders above.

You can set the DNS at docker run time with the --dns option:

$ sudo docker run --dns xx.yy.zz.tt1 --dns xx.yy.zz.tt2 -ti mmoy/ubuntu-netutils bash
root@6c5d08df5dfd:/# ping www.example.com
PING www.example.com (93.184.216.34) 56(84) bytes of data.
64 bytes from 93.184.216.34: icmp_seq=1 ttl=54 time=86.6 ms
64 bytes from 93.184.216.34: icmp_seq=2 ttl=54 time=86.6 ms

One advantage of this solution is that there is no configuration file involved, hence no risk of forgetting about the configuration and running into troubles later because of a specific config: you're getting this DNS configuration if and only if you type the --dns option.

Alternatively you may set it permanently in Docker's configuration file, /etc/docker/daemon.json (create it, on the host, if it doesn't exist):

$ cat /etc/docker/daemon.json
{
    "dns": ["xx.yy.zz.tt1", "xx.yy.zz.tt2"]
}

You need to restart the docker daemon to take the daemon.json file into account:

sudo service docker restart

Then you can check the configuration:

$ sudo docker run -ti mmoy/ubuntu-netutils bash
root@56c74d3bd94b:/# cat /etc/resolv.conf 
nameserver xx.yy.zz.tt1
nameserver xx.yy.zz.tt2
root@56c74d3bd94b:/# ping www.example.com
PING www.example.com (93.184.216.34) 56(84) bytes of data.
64 bytes from 93.184.216.34: icmp_seq=1 ttl=54 time=86.5 ms

Note that this hardcodes the DNS IP in your configuration files. This is strongly discouraged if your machine is a laptop that connects to different networks, and may be problematic if your internet service provider changes the IP of the DNS servers.

Answer 2

A clean solution is to configure docker+dnsmasq so than DNS requests from the docker container are forwarded to the dnsmasq daemon running on the host.

For that, you need to configure dnsmasq to listen to the network interface used by docker, by adding a file /etc/NetworkManager/dnsmasq.d/docker-bridge.conf:

$ cat /etc/NetworkManager/dnsmasq.d/docker-bridge.conf
listen-address=172.17.0.1

Then restart network manager to have the configuration file taken into account:

sudo service network-manager restart

Once this is done, you can add 172.17.0.1, i.e. the host's IP address from within docker, to the list of DNS servers. This can be done either using the command-line:

$ sudo docker run -ti --dns 172.17.0.1 mmoy/ubuntu-netutils bash
root@7805c7d153cc:/# ping www.example.com
PING www.example.com (93.184.216.34) 56(84) bytes of data.
64 bytes from 93.184.216.34: icmp_seq=1 ttl=54 time=86.6 ms

... or through docker's configuration file /etc/docker/daemon.json (create it if it doesn't exist):

$ cat /etc/docker/daemon.json                      
{
  "dns": [
    "172.17.0.1",
        "8.8.8.8",
        "8.8.4.4"
  ]
}

(this will fall back to Google's public DNS if dnsmasq fails)

You need to restart docker to have the configuration file taken into account:

sudo service docker restart

Then you can use docker as usual:

$ sudo docker run -ti mmoy/ubuntu-netutils bash
root@344a983908cb:/# ping www.example.com
PING www.example.com (93.184.216.34) 56(84) bytes of data.
64 bytes from 93.184.216.34: icmp_seq=1 ttl=54 time=86.3 ms

Answer 3

One way is to use a user defined network for your container. In that case the container's /etc/resolv.conf will have the nameserver 127.0.0.11 (a.k.a. the Docker's embedded DNS server), which can forward DNS requests to the host's loopback address properly.

$ cat /etc/resolv.conf
nameserver 127.0.0.1
$ docker run --rm alpine cat /etc/resolv.conf
nameserver 8.8.8.8
nameserver 8.8.4.4
$ docker network create demo
557079c79ddf6be7d6def935fa0c1c3c8290a0db4649c4679b84f6363e3dd9a0
$ docker run --rm --net demo alpine cat /etc/resolv.conf
nameserver 127.0.0.11
options ndots:0    

If you use docker-compose, it will set up a custom network for your services automatically (with a file format v2+). Note, however, that while docker-compose runs containers in a user-defined network, it still builds them in the default network. To use a custom network for builds you can specify the network parameter in the build configuration (requires file format v3.4+).

Answer 4

A brutal and unsafe solution is to avoid containerization of the network, and use the same network on the host and on the container. This is unsafe because this gives access to all the network resources of the host to the container, but if you do not need this isolation this may be acceptable.

To do so, just add --network host to the command-line, e.g.

$ sudo docker run -ti --network host mmoy/ubuntu-netutils /bin/bash
root@ubuntu1604:/# ping www.example.com
PING www.example.com (93.184.216.34) 56(84) bytes of data.
64 bytes from 93.184.216.34: icmp_seq=1 ttl=55 time=86.5 ms
64 bytes from 93.184.216.34: icmp_seq=2 ttl=55 time=86.5 ms

Answer 5

Since dnsmasq is the issue, one option is to disable it on the host. This works, but will disable DNS caching for all applications running on the host, hence is a really bad idea if the host is used for applications other than docker.

If you're sure you want to go this way, uninstall dnsmasq, e.g. on Debian-based systems like Ubuntu, run apt remove dnsmasq.

You may then check that /etc/resolv.conf within the container points to the DNS server used by the host.