Problems with Docker Network Bridge

Problem

#

TL;DR

#

By default, containers launched without a specific network configuration will create a bridge and therefore a subnet (per stack).
This can cause issues when these networks are used elsewhere.

Here we’ll see how to take this control back.

Blabla

#

I noticed that for a while now, when I connect via VPN, I cannot reach one of my machines.
However, I can reach it by going through another machine (by hopping)…
So this is a routing problem.

In the VPN, I use WireGuard from my Freebox which exclusively gives me IPs on the network 192.168.27.64/27.
But on that particular machine, there’s a bridge using address 192.168.16.1/20 !

Because of that, a packet destined for my WireGuard network will not leave this server because I already have a local route to that network !

ip route

[…]
192.168.16.0/20 dev br‑75064ab12d5d proto kernel scope link src 192.168.16.1
[…]

The culprit is Docker, which seems to use by default the following subnets for these bridges:

• 172.17.[0‑31].1/16
• 192.168.[n*16].1/20 (192.168.[0,16,32,…,240].1/20)

dockerd is smart enough to avoid networks already used on the host but doesn’t go further.

Experimentation

#

Base

#

To reproduce my production environment as closely as possible, I set up an Ubuntu 20.04 LTS VM on my PC (VirtualBox).

I installed Docker and the Docker Compose plugin so that I could create multiple containers to simulate several applications running at the same time.

For this, I chose to run Nginx containers. This way, it will be easy to verify the services’ functionality using a web browser.

sudo -i

Here’s how I generated 40 Nginx containers with Docker Compose :

mkdir -pv /srv/dockers
cd /srv/dockers

for i in $(seq 8080 8120); do \
  echo $i; \
  mkdir -pv ${i}; \
  echo "services:" > ./${i}/compose.yml; \
  echo "  nginx_${i}:" >> ./${i}/compose.yml; \
  echo "    image: nginx:latest" >> ./${i}/compose.yml; \
  echo "    container_name: nginx_${i}" >> ./${i}/compose.yml; \
  echo "    restart: unless-stopped" >> ./${i}/compose.yml; \
  echo "    ports:" >> ./${i}/compose.yml; \
  echo "      - ${i}:80" >> ./${i}/compose.yml; \
done

Test 1

#

Let’s just launch all containers and see what happens :

cd /srv/dockers
for i in $(seq 8080 8120); do \
  cd $i; \
  pwd; \
  docker compose up -d; \
  cd ..; \
done

We see that it blocks when creating the network 8109_default because there are no more subnets available via Docker daemon’s default configuration (dockerd).

To view the interfaces:

ip a | grep "br-" | grep inet

Before moving on, we destroy all containers :

cd /srv/dockers
for i in $(seq 8080 8120); do \
  cd $i; \
  pwd; \
  docker compose down; \
  cd ..; \
done

Test 2

#

The Docker documentation shows parameters we can use to control our bridge.

So I set them up :

cat << EOF > /etc/docker/daemon.json
{
  "bip": "10.20.1.1/16",
  "fixed-cidr": "10.20.1.0/17"
}
EOF

And apply it :

systemctl restart docker

After running ip a show docker0, we see that our bridge docker0 has taken the new address.

Now, we can relaunch our containers but let’s do it with just one for now :

cd /srv/dockers/8080/
docker compose up -d

Looking at the IP of the newly created bridge :

ip a | grep "br-" | grep inet

inet 172.17.0.1/16 brd 172.17.255.255 scope global br-9eaa90793383

We realize it didn’t use the network we described in the configuration file.

Curiously, I ran a few tests that all worked :

curl http://127.0.0.1:8080
curl http://10.20.1.1:8080
curl http://172.17.0.1:8080

So this was the wrong path; it didn’t solve our problem…

We shut down the service before starting a new test :

cd /srv/dockers/8080/
docker compose down

Test 3

#

I couldn’t find anything convincing in Docker’s documentation to fix my issue.
However, I found mention of the default-address-pools parameter on
their forum.

So let’s try that!

cat << EOF > /etc/docker/daemon.json
{
    "default-address-pools": [
        { "base":"10.20.0.0/16","size":24 }
    ]
}
EOF

systemctl restart docker
ip a show docker0

Strangely, the default bridge didn’t change address and stayed at 10.20.0.1/24…

We then restarted all containers:

cd /srv/dockers
for i in $(seq 8080 8120); do \
  cd $i; \
  pwd; \
  docker compose up -d; \
  cd ..; \
done

Now they all start fine ! All networks are present (ip a | grep "br-" | grep inet).

I conclude that setting the address pool based on 10.20.0.0/16 with size 24 gives each bridge a /24 subnet inside the /16.
So bridges will use subnets like 10.20.[0‑255].0/24, which is enough to create 256 (8 bits between /16 and /24) bridges with up to 254 containers per stack!

Application

#

On my production server, I replicate the same steps :

I bring down all my containers :

cd /srv/dockers
for i in $(ls); do \
  cd $i; \
  pwd; \
  docker compose down; \
  cd ..; \
done

Then I create the file (which, for me, doesn’t exist) :

cat << EOF > /etc/docker/daemon.json
{
  "bip": "10.20.1.1/16",
  "fixed-cidr": "10.20.1.0/17"
}
EOF

systemctl restart docker

cd /srv/dockers
for i in $(ls); do \
  cd $i; \
  pwd; \
  docker compose up -d; \
  cd ..; \
done

We check:

ip a | grep "br-" | grep inet

And it’s OK!

Conclusion

#

So we’ve seen how to change the IP address of the docker0 bridge and those created dynamically.

I don’t understand why this isn’t better documented in the official docs.
After some research, I found it
here,
in an IPv6 section where I never thought to look.
It’s really unfortunate that it’s not easier to find…

So, that solved my VPN‑through‑access problem !

Hope this helps someone ! 😉