Feature: Flannel integration

[leon3s]

Is your feature request related to a problem? Please describe.
To enable communication between container in a multi node setup, we need to be able to integrate with flannel:

This is what is discovered so far with my testing:

First we will start 2 container with the docker:dnd image:

docker run -it --name node1 --privileged docker:dind
docker run -it --name node1 --privileged docker:dind

We will assume for the rest of the tutorial that node1 have 172.17.0.2 as ip address
and node2 172.17.0.3, this may change depending on your docker setup you can inspect node1 and node2 to get their ip addresse.

Then on the first node we initialize an etcd instance:

docker run -d \
  -p 2379:2379 \
  -p 2380:2380 \
  --name etcd \
  --restart always \
  quay.io/coreos/etcd:v3.4.13 \
  /usr/local/bin/etcd \
  --name etcd1 \
  --data-dir /var/lib/etcd \
  --listen-client-urls http://0.0.0.0:2379 \
  --advertise-client-urls http://172.17.0.2:2379 \
  --listen-peer-urls http://0.0.0.0:2380 \
  --initial-advertise-peer-urls http://172.17.0.2:2380 \
  --initial-cluster-token etcd-cluster-1 \
  --initial-cluster my-etcd-1=http://172.17.0.2:2380 \
  --initial-cluster-state new --enable-v2

We need to add a new member for the etcd cluster to prepare the 2nd node:

etcdctl --endpoints=http://172.17.0.2:2379 member add etcd2 --peer-urls=http://172.17.0.3:2380

Then on the node2 we can start the 2nd etcd instance:

docker run -d \
  --name etcd2 \
  -p 2379:2379 \
  -p 2380:2380 \
  quay.io/coreos/etcd:v3.4.13 \
  /usr/local/bin/etcd \
  --name etcd2 \
  --initial-advertise-peer-urls http://172.17.0.3:2380 \
  --listen-peer-urls http://0.0.0.0:2380 \
  --listen-client-urls http://0.0.0.0:2379 \
  --advertise-client-urls http://172.17.0.3:2379 \
  --initial-cluster etcd1=http://172.17.0.2:2380,etcd2=http://172.17.0.3:2380 \
  --initial-cluster-state existing --enable-v2

Now we can start flannel !

On both node:

docker run -d \
  --name flannel \
  --privileged \
  --network host \
  --volume /run/flannel:/run/flannel \
  --volume /lib/modules:/lib/modules \
  quay.io/coreos/flannel:v0.15.1 \
  /opt/bin/flanneld --etcd-endpoints=http://localhost:2379

We can check if everything is working by loggin the containers.

Now that we have etcd and flannel up and running we can create docker network:

First we need to get the subnet that flannel choosed for the node, to do so we can cat /run/flannel/subnet.env

For my first node it show 10.244.40.1/24

So we can create the docker network on the first node as follow:

docker network create \
  --subnet=10.244.40.0/24 \
  --opt com.docker.network.bridge.name=flannel1 \
  flannel-net1

On the second node i have a different subnet, dont forget to cat /run/flannel/subnet.env to get the values
10.244.82.1/24

docker network create \
  --subnet=10.244.82.0/24 \
  --opt com.docker.network.bridge.name=flannel1 \
  flannel-net1

Now we can create container on both node:

docker run --it --network flannel-net1 busybox:latest

And you should be able to ping them in both way!

While this solution seems good, after further research we only have 256 IPv4 addresse available for the network, meaning we can't really scale well, we should use ipv6 by default to be able to scale well the number of instance available in one node.

Flannel doesn't seems to support ipv6 so we should take a look into calico.
But having a flannel support will be nice in the first place

[unixfox]

It would also allow to use wireguard in a distributed way. Which is cool: https://github.com/flannel-io/flannel/blob/master/Documentation/backends.md#wireguard

[leon3s]

Yeah but be aware that it may cause network performance issue.
I'll recommend to use end to end SSL/TLS between your services even for internal communications