Running an etcd 2.0 Cluster on Ubuntu 14.0415 April 2015 · Filed in Tutorial
In this post, I’m going to show you how to set up a cluster of three nodes running etcd 2.0 (specifically, etcd 2.0.9). While I’ve discussed etcd before, that was in the context of using etcd with CoreOS Linux. In this case, I’ll use Ubuntu 14.04 as the base OS, along with the latest released version of etcd.
To help you follow along, I’ve created a set of files that will allow you to use Vagrant to turn up an etcd 2.0 cluster on Ubuntu 14.04 (on your laptop, if so desired). You can find all these files in the “etcd-2.0” directory of my learning-tools GitHub repository.
Installing the Base OS
You don’t need anything special when setting up etcd; a straightforward Ubuntu Server 14.04 x64 installation will work just fine. If you’re using the files in my learning-tools repository, you’ll see that Vagrant simply turns up a VM based on a plain-jane Ubuntu 14.04 box. If you’re building this from scratch (why?!), simply create a VM and install Ubuntu 14.04 into it. As long as it has Internet connectivity, that’s all that’s needed.
Installing etcd is very straightforward:
Download the etcd 2.0.9 release from https://github.com/coreos/etcd/releases/tag/v2.0.9.
Unpack the downloaded
tar xvzf <filename>.
/var/etcddirectory for etcd to use as its data directory.
Optionally, change into the newly-created directory (named
etcd-2.0.9-linux-amd64by default) and use
mvto move the
etcdctlbinaries to a more permanent location on your system. (These are statically linked binaries, so feel free to put them wherever it makes sense to you. I chose to put them in
That’s it. Just make a note of where the
etcd binary is stored, as you’ll need to know that location later.
Configuring etcd and Turning Up the Cluster
Earlier (pre-2.0) releases of etcd apparently supported a TOML-based configuration file, but that support was removed in the 2.0 release. Instead, you’ll need to use either a) environment variables or b) command-line parameters. I chose to use environment variables, and—since I wanted to make this as “production quality” as possible, I chose to build an Upstart script for etcd. (If you don’t want to use Upstart, I’ll talk about that later in this section.)
Here’s an Upstart script you can use for etcd:
This is a reasonably straightforward Upstart script, so I won’t bother spending a great deal of time explaining it. Note that if you chose not to move the
etcd binary to
/usr/local/bin, you’ll have to edit this Upstart script to point to the correct location of the binary. Also, if you choose to use something other than
/var/etcd as the data directory, you’ll need to edit this Upstart script appropriately. Consider the script above to be an example from which you can build one appropriate to your environment.
I did find that the
if [ -f "/etc/default/etcd" ]; then test—and subsequently sourcing the contents of that file—doesn’t work with Upstart (they should be removed from this Upstart script, but I haven’t bothered yet). The original idea was to define the necessary environment variables in
/etc/default/etcd, but no amount of effort could make it work. I could embed the environment variables in the Upstart script itself, but that seemed like a hack (and not a good one). Finally, a couple folks pointed me to override files, which worked perfectly.
The Upstart script itself (listed above) goes into
etcd.conf. The override file also goes into
/etc/init but is named (quite intuitively)
etcd.override. Here’s the contents of the override file:
By including all these environment variables in the override file, we’re able to keep the Upstart script itself very clean and minimal. If you need to make changes to the configuration, then you edit the override file and just restart the service. Without the override file, all these parameters would have to be specified on the
etcd command line itself, which has the following effects:
- You can’t use the same Upstart script on multiple systems, since the script contains configuration data specific to a particular node (like the hostname and IP addresses).
- The Upstart script becomes unnecessarily complex and harder to debug and/or troubleshoot.
Moving the environment variables into the override file solves both these problems. Machine-specific override files can be supplied for each system and used with a single, consistent Upstart script.
If you’re not really familiar with etcd, the environment variables defined in the override file deserve a bit of explanation. In this configuration, etcd is using static discovery to build the cluster. In other words, you (the user) are explicitly providing the hostnames and/or IP addresses of all of the members of the cluster (specified in the
ETCD_INITIAL_CLUSTER variable). As described here, the
ETCD_INITIAL_* variables are only used on the first run of etcd; they will be ignored on subsequent runs. This allows you to easily bootstrap an etcd cluster, if you know the IP addresses that will be used in advance. (By the way, this video on bootstrapping etcd is great—highly recommended.) Naturally, the IP addresses specified in the override file for all the etcd nodes need to be correct.
If you choose not to use Upstart and just launch
etcd manually, then you’ll either need to a) manually define the appropriate environment variables, or b) specify those configuration parameters on the command line you use to launch
Assuming that you’ve chosen to use Upstart and have correctly configured the override file on each system that will be part of the cluster, you can then start etcd with a simple
sudo initctl start etcd. Some error messages will be logged in the log file (
/var/log/etcd.log) until all three nodes come online, but once all three nodes (as specified in the
ETCD_INITIAL_CLUSTER variable) are up and running etcd will be running as expected.
You can verify the operation of etcd by running a command like
etcdctl member list, which will return a list of the members of the etcd cluster. Congratulations, you’ve just turned up an etcd 2.0 cluster!
You can find the Upstart script, machine-specific override files, a provisioning script, and a Vagrantfile for replicating a three-node etcd 2.0 cluster on Ubuntu 14.04 in the “etcd-2.0” directory of my learning-tools GitHub repository. It was tested with Vagrant 1.7.2, the VMware plugin for Vagrant, and Fusion 6.0.5. It should work on VMware Workstation, but I haven’t tested it there. (Feedback is welcome.)Tags: CLI · Fusion · Linux · Macintosh · Ubuntu · Vagrant Previous Post: The Plain Text Productivity Experiment Next Post: Running an etcd-Backed Docker Swarm Cluster