Experimenting with Docker, Registrator, and Consul

 Published on 28 Feb 2015 ·  Filed in Information ·  543 words (estimated 3 minutes to read)

Over the last few days, I’ve been experimenting with Docker, Registrator, and Consul in an effort to explore some of the challenges involved in building a robust containerized infrastructure. While I haven’t finished fully exploring the idea (and documenting what I’ve learned), I did discover one interesting—and unexpected—interaction.

Here’s a quick overview of my testing environment:

• I used two OpenStack Heat templates to spin up two clusters of 5 instances each.
• The first cluster is a set of CoreOS Linux instances, customized via cloud-init to not run etcd. These instances are attached to a VMware NSX-powered logical network using IP addresses from the 10.1.1.0/24 subnet.
• On each CoreOS Linux instance, I have Registrator running as a Docker container and listening to the Docker socket (thus listening to Docker events).
• The second cluster is a set of Ubuntu 14.04 instances running Consul. These instances are connected to an NSX-powered logical network using IP addresses from the 10.1.2.0/24 subnet.
• The two logical networks are connected by a logical router and thus have full connectivity.

Registrator, if you’re not already familiar with it, is a service registry tool that listens to the Docker socket and registers/de-registers containers with a backend registry service (like Consul). This makes it an essential part of a dynamic containerized infrastructure, as service registration and service discovery are key challenges that must be addressed.

Based on some earlier testing, what I expected to happen in this environment was this:

1. I would fire up a Docker container on one of the CoreOS instances.
2. Registrator running on that instance detects the new container and registers it with Consul (via one of the instances running Consul). The IP address registered with Consul would be the IP address of the CoreOS instance where the container is running (something from 10.1.1.0/24).
3. Services/applications/users could then discover (and access) the Docker container via the CoreOS instance’s IP address by querying Consul (either via HTTP API or via DNS).

However, what I found actually happened was this:

1. A new Docker container is launched on one of the CoreOS instances.
2. Registrator detects the new container and registers it with Consul. However, because Consul is running on a separate node, the IP address that gets registered is the IP address of the Consul node itself, not the CoreOS host instance.
3. Services/applications/users attempting to discover the new service think it resides on the Consul node (something on 10.1.2.0/24), when in reality it is running on a CoreOS instance.

I did not expect this behavior. From what I’ve been able to find (see this issue) this is expected behavior from Consul, and the only fix is to run Consul as a Docker container itself (a so-called “sidecar” container) on every host where Registrator is running. This is the configuration I used in my first round of testing, and thus why I was caught off-guard by the incorrect registration behavior in the new environment. In the next iteration of the testing environment, I’ll run Consul as a sidecar container, although I need to deal with bootstrapping the Consul cluster in an environment where CoreOS instances will automatically reboot due to system updates.

I still have more testing to do (and more learning to do), and I’ll continue to share the results of my testing/learning here.

Metadata and Navigation

 Docker  Networking  DNS
Previous Post: Enabling and Mounting NFS on CoreOS
Next Post: Running a Small Docker Swarm Cluster

Be social and share this post!