Official documentation describes the DNS service discovery configuration rather clearly. If you’re running consul as a system-wide service and running on the latest Ubuntu 20 LTS, you might encounter some gotchas as I did. I’m going to list a few for future-me and random internet strangers ;).

Systemd-resolved vs dnsmasq

Latest ubuntu 20 LTS ships with systemd older than 246. That requires mapping ports with iptables. I had very little luck with that (for whatever reason, I don’t even recall anymore 😅), so instead, I installed Dnsmasq. Remember to disable systemd-resolved after you install dnsmasq - otherwise, you might end up with a broken DNS configuration.

# once you have dnsmasq installed you can disable systemd-resolved
systemctl stop systemd-resolved
systemctl disable systemd-resolved

Dnsmasq configuration

Here is mine /etc/resolv.conf file. I prefer using Cloudflare DNS as a fallback if dnsmasq gets misconfigured or any other potential crash.

That should just work as expected according to this doc:

If there are multiple servers, the resolver library queries them in the order listed

# /etc/resolv.conf
nameserver 127.0.0.1
nameserver 1.1.1.1

To configure dnsmasq for consul I created /etc/dnsmasq.d/10-consul file. Consul DNS service is running on port 8600, dnsmasq is listening on localhost and Docker network bridge (172.17.0.1).

server=/consul/127.0.0.1#8600
bind-interfaces
no-resolv
server=1.1.1.1
server=1.0.0.1
listen-address=127.0.0.1
listen-address=172.17.0.1

Depending on how you install dnsmasq I recommend looking into /lib/systemd/system/dnsmasq.service and ensuring to always restart service on failure.

[Service]
TimeoutStartSec=0
Restart=on-failure
RestartSec=5s
...

Consul configuration

Consul DNS service listens on localhost on port 8600 - nothing fancy here.

# /etc/consul/config.json
"addresses": {
  "dns": "127.0.0.1",
  ...
},
"ports": {
  "dns": 8600,
  ...
}

Consul DNS ACL gotchas

One minor but crucial thing to notice here is that if you have acl enabled with "default_policy": "deny" in consul, your DNS queries will NOT WORK by default. The irony here is that it’s also extremely easy to miss.

To fix it first create dns-request-policy.hcl file:

node_prefix "" {
  policy = "read"
}
service_prefix "" {
  policy = "read"
}
# only needed if using prepared queries
query_prefix "" {
  policy = "read"
}

Then create a new policy in consul:

consul acl policy create -name "dns-requests" -rules @dns-request-policy.hcl

Create a token and use that token as consul default agent token:

consul acl token create -description "Token for DNS Requests" -policy-name dns-requests
consul acl set-agent-token default <token from command above>

Docker configuration

To query consul service from within the docker container, we need to define dns servers in /etc/docker/daemon.json. We will point the first one to the docker bridge network and add a fallback to Cloudflare just in case.

Honestly speaking, I’m not 100% sure this is still required as according to this docs all external DNS lookups will be forwarded to the host by default:

By default, a container inherits the DNS settings of the host, as defined in the /etc/resolv.conf configuration file. Containers that use the default bridge network get a copy of this file, whereas containers that use a custom network use Docker’s embedded DNS server, which forwards external DNS lookups to the DNS servers configured on the host.

In case you would like to configure it explicitly regardless, you can just do:

# /etc/docker/daemon.json
{
  "dns": [
    "172.17.0.1",
    "1.1.1.1"
  ],
  ...
}

Once everything is up and running you should be able to query your consul services from within the docker containers via <service-name>.service.consul address.