CLI

You are currently browsing articles tagged CLI.

I’ve written quite a bit about Open vSwitch (OVS), but I realized recently that despite all the articles I’ve written I still haven’t talked about how to remove a configuration setting to OVS. I’m fixing that now with this article.

As part of my ongoing mission to give back to the open source community, I recently started making contributions and improvements to the OVS web site; specifically, I’ve been reformatting the configuration cookbooks to make them more readable (and to clean up the HTML source). Along the way, I’ve been adding small bits of content here and there. Most recently, I just updated the QoS rate-limiting entry, and I wanted to add information on how to remove the QoS settings.

Normally, you can remove an OVS configuration setting using the ovs-vsctl remove command. For example, if you set a VLAN tag on an port with this command:

ovs-vsctl set port vnet0 tag=100

Then you could remove that VLAN tag with this command:

ovs-vsctl remove port vnet0 tag 100

Note the slight syntactical difference in the two commands; the remove command expects four parameters.

It turns out, however, that this command won’t work for all configuration parameters. In some cases, you can use the ovs-vsctl remove command, but in some cases you have to use the ovs-vsctl set command to set it back to the original value. (This is how you “remove” QoS rate-limiting, by the way: you set the QoS values back to 0.)

How does one know when to use one command versus the other? The answer lies within the Open vSwitch database schema. (This document is your friend.)

If you look through the OVS database schema, you’ll find that some values (the document references them as “columns”) will be listed as “integers,” while others are listed as “optional integers.” For example, the tag column in the Port table (which is where VLAN tags are stored for an OVS port) is listed as an optional integer. The ingress_policing_rate column in the Interface table (where QoS rate-limiting is specified) is listed as an integer.

Why is that important? Here’s why:

  • If column is classified as an optional integer, then you can use ovs-vsctl remove to remove the value.
  • If, on the other hand, a column is specified as an integer, then you have to set the value back to its original value (using ovs-vsctl set) to remove the value. You can’t use ovs-vsctl remove as it will generate an error.

I hope this makes sense. Feel free to hit me up if you have any questions, or if anything I’ve shared here is incorrect or inaccurate.

Tags: , , ,

For non-programmers, making a meaningful contribution to an open source project can be difficult; this is as true for OpenStack as for other open source projects. Documentation is a way to contribute, but in the case of OpenStack there is a non-trivial setup required in order to be able to contribute to the OpenStack documentation. In this post, I’m going to share how to set up the tools to contribute to OpenStack documentation in the hopes that it will help others get past the “barrier to entry” that currently exists.

I’ve long wanted to be more involved in supporting the OpenStack community, beyond my unofficial support via advocacy and blogging about OpenStack. I felt that documentation might be a way to achieve that goal. After all, I’ve written books and have been blogging for 9 years, so I should be able to add some value via documentation contributions. However, the toolchain that the OpenStack documentation uses requires a certain level of familiarity with development-focused tools, and the “how to” guides were less than ideal because of assumptions made regarding the knowledge level of new contributors. For these reasons, I felt that sharing how I (a non-programmer) set up the tools for contributing to OpenStack documentation might encourage other non-programmers to do the same, and thus get more people involved in the project.

This post is not intended to replace any official guides (like this one or this one), but rather to supplement such guides.

Using a Linux Environment

It’s no secret that I use OS X, and the toolchain that the OpenStack documentation team uses is—like the rest of OpenStack—pretty heavily biased toward Linux. One of the deterring factors for me was the level of difficulty to get these tools running on OS X. While it is possible to use the toolchain on OS X directly, it’s not necessarily simple or straightforward. It wasn’t until just recently that I realized: why not just use a Linux VM? Using a native Linux environment sidesteps all the messiness of trying to get a Linux-centric toolchain running on OS X (or Windows).

I initially started with a locally-hosted Linux VM on my MacBook Air, but after some consideration I decided to alter that approach and build my environment in a cloud VM running Ubuntu 12.04 on Amazon EC2 (via Ravello Systems).

The use of a cloud VM has some advantages and disadvantages:

  • Because the toolchain is heavily biased toward Linux, using a Linux-based cloud VM does simplify some things.
  • Cloud VMs will typically have much greater bandwidth, making it easier to install packages and pull down Git repositories. This was especially helpful this past week while I was traveling to the OpenStack Summit, where—due to slow conference wireless access—it took 2 hours to clone the GitHub repository for the OpenStack documentation (this is while I was trying to use a locally-hosted VM on my laptop).
  • It provides a clean separation between my local workstation and the development environment, meaning I can access my development environment from any system via SSH. This prevents me from having to keep multiple development environments in sync, which is something I would have had to do when working from my home office (where I use my MacBook Air as well as a Mac Pro).
  • The most significant drawback is that I cannot work on documentation patches when I have no network connectivity. For example, I’m writing this post as I’m traveling back from Paris. If I had a local development environment, I could assigned a few bugs to myself and worked on them while flying home. With a cloud VM-based environment, I cannot. I could have used a VM hosted on my local laptop (via VirtualBox or VMware Fusion), but that would have negated some of the other advantages.

In the end, you’ll have to evaluate for yourself what approach works best for you, your working style, and your existing obligations. However, the use of a Linux VM—local or via a cloud provider—can simplify the process of setting up the toolchain and getting started contributing, and it’s an approach I’d recommend for most anyone.

The information presented here assumes you are using an Ubuntu-based VM, either hosted with a cloud provider or on your local system. I’ll leave it to the readers to worry about the details of how to turn up the Ubuntu VM via their preferred mechanism.

Setting Up the Toolchain

Once you’ve established an Ubuntu Linux instance using the provider of your choice, use these steps to set up the documentation toolchain (note that, depending on the configuration of your Ubuntu Linux instance, you might need to preface these commands with sudo where appropriate):

  1. Update your system using apt-get update and apt-get -y upgrade. Depending on the updates that are installed, you might need to reboot.

  2. Generate an SSH keypair using ssh-keygen. Keep track of this keypair (and make note of the associated password, if you assigned one), as you’ll need it later. You can also re-use an existing keypair, if you’d like; I preferred to generate a keypair specifically for this purpose. If you are going to re-use an existing keypair, you’ll need to transfer them to this Linux instance via scp or your preferred mechanism.

  3. Install the necessary prerequisite packages (I’ve line wrapped the command below, denoted by a backslash, for readability):

    apt-get install maven git git-review python-pip libxml2-dev \
    libxslt1-dev python-dev gcc gettext zlib1g-dev
  4. Configure Git using git config to set your name and e-mail address. The e-mail address, in particular, should match what you’re using with the OpenStack Foundation. The commands to do this are git config --global
    user.name
    and git config --global user.email.

  5. Clone the GitHub repository where the documentation is found:

    git clone https://github.com/openstack/openstack-manuals.git
  6. Install the components required to run tests against the documentation sources (this command assumes that you cloned the repository into a subdirectory named “openstack-manuals”, which would be the default result of using the command above to clone the repository):

    pip install -r openstack-manuals/test-requirements.txt
  7. If you haven’t already established a Launchpad account, you’ll need one. Go create one (probably best done outside of the Ubuntu Linux environment) and make note of the username. Use your Launchpad account to verify that you can log in to both bugs.launchpad.net as well as review.openstack.org.

  8. While logged into review.openstack.org, go to the settings for your account and paste in the contents of the public key (not the private key) you generated earlier (or that you are re-using).

  9. Back in the Ubuntu Linux instance, set your Launchpad username:

    git config --global --add gitreview.username "<Launchpad username>"
  10. Change into the directory where you cloned the “openstack-manuals” repository and run git review -s. Assuming you already added the “gitreview.username” configuration parameter and that you’ve already uploaded the public key to your account on review.openstack.org, this should work without any issues. If you forgot to set “gitreview.username”, it should prompt you for your Launchpad username. If you forgot to upload the public SSH key, then it will probably error out.

  11. If git review -s completes without any obvious errors, you can double-check that a Gerrit upstream for the repository was added by using the command git remote -v. If you don’t see an upstream repo labeled gerrit, then something didn’t work. If you do see the upstream repo labeled gerrit, then you should be fine.

Once you’ve completed these steps, then you’re ready to start contributing to OpenStack documentation. I have a separate post planned that describes the process for actually contributing; stay tuned for that soon.

In the meantime, if anyone has any questions, corrections, or clarifications about the information in this post, please speak up in the comments below.

Tags: , , , , ,

This post will provide a quick introduction to a tool called Vagrant. Unless you’ve been hiding under a rock—or, more likely, been too busy doing real work in your data center to pay attention—you’ve probably heard of Vagrant. Maybe, like me, you had some ideas about what Vagrant is (or isn’t) and what it does (or doesn’t) do. Hopefully I can clear up some of the confusion in this post.

In its simplest form, Vagrant is an automation tool with a domain-specific language (DSL) that is used to automate the creation of VMs and VM environments. The idea is that a user can create a set of instructions, using Vagrant’s DSL, that will set up one or more VMs and possibly configure those VMs. Every time the user uses the precreated set of instructions, the end result will look exactly the same. This can be beneficial for a number of use cases, including developers who want a consistent development environment or folks wanting to share a demo environment with other users.

Vagrant makes this work by using a number of different components:

  • Providers: These are the “back end” of Vagrant. Vagrant itself doesn’t provide any virtualization functionality; it relies on other products to do the heavy lifting. Providers are how Vagrant interacts with the products that will do the actual virtualization work. A provider could be VirtualBox (included by default with Vagrant), VMware Fusion, Hyper-V, vCloud Air, or AWS, just to name a few.
  • Boxes: At the heart of Vagrant are boxes. Boxes are the predefined images that are used by Vagrant to build the environment according to the instructions provided by the user. A box may be a plain OS installation, or it may be an OS installation plus one or more applications installed. Boxes may support only a single provider or may support multiple providers (for example, a box might only work with VirtualBox, or it might support VirtualBox and VMware Fusion). It’s important to note that multi-provider support by a box is really handled by multiple versions of a box (i.e, a version supporting VirtualBox, a version supporting AWS, or a version supporting VMware Fusion). A single box supports a single provider.
  • Vagrantfile: The Vagrantfile contains the instructions from the user, expressed in Vagrant’s DSL, on what the environment should look like—how many VMs, what type of VM, the provider, how they are connected, etc. Vagrantfiles are so named because the actual filename is Vagrantfile. The Vagrant DSL (and therefore Vagrantfiles) are based on Ruby.

Once of the first things I thought about as I started digging into Vagrant was that Vagrant would be a tool that would help streamline moving applications/code from development to production. After all, if you had providers for Vagrant that supported both VirtualBox and VMware vCenter, and you had boxes that supported both providers, then you could write a single Vagrantfile that would instantiate the same environment in development and in production. Cool, right? In theory this is possible, but in talking with some others who are much more familiar with Vagrant than I am it seems that in practice this is not necessarily the case. Because support for multiple providers is handled by different versions of a box (as outlined above), the boxes may be slightly different and therefore may not produce the exact same results from a single Vagrantfile. It is possible to write the Vagrantfile in such a way as to recognize different providers and react differently, but this obviously adds complexity.

With that in mind, it seems to me that the most beneficial uses of Vagrant are therefore to speed up the creation of development environments, to enable version control of development environments (via version control of the Vagrantfile), to provide some reasonable level of consistency across multiple developers, and to make it easier to share development environments. (If my conclusions are incorrect, please speak up in the comments and explain why.)

OK, enough of the high-level theory. Let’s take a look at a very simple example of a Vagrantfile:

(Click here if you can’t see the code block above.)

This Vagrantfile sets the box (“ubuntu/precise64″), the box URL (retrieves from Canonical’s repository of cloud images), and then sets the “/vagrant” directory in the VM to be shared/synced with the current (“.”) directory on the host—in this case, the current directory is the directory where the Vagrantfile itself is stored.

To have Vagrant then use this set of instructions, run this command from the directory where the Vagrantfile is sitting:

vagrant up

You’ll see a series of things happen; along the way you’ll see a note that the machine is booted and ready, and that shared folders are getting mounted. (If you are using VirtualBox and the box I’m using, you’ll also see a warning about the VirtualBox Guest Additions version not matching the version of VirtualBox.) When it’s all finished, you’ll be deposited back at your prompt. From there, you can easily log in to the newly-created VM using nothing more than vagrant ssh. That’s pretty handy.

Other Vagrant commands include:

  • vagrant halt to shut down the VM(s)
  • vagrant suspend to suspend the VM(s), use vagrant resume to resume
  • vagrant status to display the status of the VM(s)
  • vagrant destroy to destroy (delete) the VM(s)

Clearly, the example I showed you here is extremely simple. For an example of a more complicated Vagrantfile, check out this example from Cody Bunch, which sets up a set of VMs for running OpenStack. Cody and his co-author Kevin Jackson also use Vagrant extensively in their OpenStack Cloud Computing Cookbook, 2nd Edition, which makes it easy for readers to follow along.

I said this would be a quick introduction to Vagrant, so I’ll stop here for now. Feel free to post any questions in the comments, and I’ll do my best to answer them. Likewise, if there are any errors in the post, please let me know in the comments. All courteous comments are welcome!

Tags: , ,

CoreOS Continued: Fleet and Docker

This post is the third in a series of posts on CoreOS, this time focusing on the use of fleet and Docker to deploy containers across a cluster of systems. This post builds on my earlier introduction to CoreOS and the subsequent more in-depth look at etcd.

I’m assuming that you’re already reasonably familiar with CoreOS, etcd, and Docker. If you aren’t familiar with CoreOS or etcd, have a look at the links in the previous paragraph. If you need a quick introduction to Docker, check out this quick introduction to Docker. While the example I’m going to provide here is fairly simple, it should serve as a reasonable basis upon which to build later.

An Overview of Fleet

The GitHub page for fleet describes it as a “distributed init system” that operates across a cluster of machines instead of on a single machine. It leverages etcd, the distributed key-value store that ships with CoreOS, as well as systemd. Fleet combines etcd and systemd to allow users to deploy containers (configured as systemd units) across a cluster of CoreOS systems.

Using fleet, users can deploy a single container anywhere on the cluster, deploy multiple copies of the same container, ensure that containers run on the same machine (or different machines), or maintain a certain number of instances of a service (thus protecting against failure).

Note that even though fleet helps with scheduling containers across a cluster of systems, fleet doesn’t address some of the other significant challenges that arise from an architecture based on distributed micro-services in containers. Namely, fleet does not address inter-container communications, service registration, service discovery, or any form of advanced utilization-based scheduling. These are topics I hope to be able to explore here in the near future.

Now that you have an idea of what fleet does, let’s take a closer look at actually using fleet.

Interacting with Fleet

By default, the fleetctl command-line client that is provided to interact with fleet assumes it will be interacting with a local etcd endpoint on the loopback address. So, if you want to run fleetctl on an instance of CoreOS in your cluster, no further configuration is needed.

However, it may be easier/more effective to use fleetctl from outside the cluster. There are a couple of different ways to do this: you can tell fleetctl to use a specific endpoint, or you can tunnel the traffic through SSH. Each approach has advantages and disadvantages; I’ll leave it to the readers to determine which approach is the best approach for their specific configurations/situations.

Using a Custom Endpoint

This method is pretty straightforward and simple. Just set an environment variable named FLEETCTL_ENDPOINT, like this:

export FLEETCTL_ENDPOINT=http://10.1.1.7:4001

Obviously, you’d want to make sure that you have the correct IP address (can be any node in the etcd cluster) and port (4001 is the default, I believe). With this environment variable set, now anytime you use fleetctl it will direct traffic to the endpoint you specified. If that specific node in the etcd cluster becomes unavailable, then fleetctl will stop working, and you’ll need to point it to a different node in the cluster.

Tunneling Through SSH

The second way of using fleetctl remotely is to tunnel the traffic through SSH. This method may be a bit more complicated, but naturally offers a bit more security.

To make fleetctl tunnel its communications with etcd through SSH, set an environment variable called FLEETCTL_TUNNEL to the IP address of any node in the etcd cluster, like this:

export FLEETCTL_TUNNEL=10.1.1.7

However, the configuration involves more than just setting the environment variable. The fleetctl doesn’t expose any options to configure the SSH connection, and it assumes you’ll be using public key authentication. This means you’ll need access to a public key that will work against the nodes in your etcd cluster. If you followed my instructions on deploying CoreOS on OpenStack via Heat, then you can review the Heat template to see which key was specified to be injected when the instances were spawned. Once you know which key was used, then you’ll need to either:

  • place that key on the system where fleetctl is installed, or
  • install fleetctl on a system that already has that key present.

There’s still at least one more step required (possibly two). Because fleetctl doesn’t expose any SSH options, you’re going to need to run an SSH agent on the system you’re using. OS X provides an SSH agent by default, but on Linux systems you will probably have to manually run an SSH agent and add the appropriate SSH key:

eval `ssh-agent -s`
ssh-add ~/.ssh/keyfile.pem

Once the SSH agent is running and the appropriate key is loaded (you’d clearly need to make sure the path and filename are correct in the command listed above), then the last step is to configure your ~/.ssh/config file with options for the CoreOS instances. It’s possible you might be able to get by without this step; I haven’t conducted enough testing to say with absolute certainty one way or another. I suspect it will be needed.

In the ~/.ssh/config file, add a stanza for the system through which you’ll be tunneling the fleetctl traffic. The stanza will need to look something like this:

Host node-01
  User core
  Hostname 10.1.1.7
  IdentityFile ~/.ssh/keyfile.pem

This configuration stanza ensures that when the system you’re using attempts to communicate with the IP address listed above, it will use the specified username and public key. Since the SSH agent is loaded, it won’t prompt for any password for the public key (even if the public key doesn’t have a password associated, you’ll still need the SSH agent), and the SSH connection will be successful without any user interaction. That last point is important—fleetctl doesn’t expose any SSH options, so the connection needs to be completely automatic.

Once you have all these pieces in place, then you can simply run fleetctl with the appropriate commands (described in the next section), and the connection to the etcd cluster will happen over SSH via the specified host. Naturally, if that node in the cluster goes away or is unavailable, you’ll need to point your connection to a different node in the etcd cluster.

Using Fleet

Once you have access to the etcd cluster via fleetctl using one of the three methods described above (direct access via a CoreOS instance, setting a custom endpoint, or tunneling over SSH), then you’re ready to start exploring how fleet works.

First, you can list all the machines in the cluster with this command:

fleetctl list-machines

Note the “METADATA” column; this allows you to do some custom scheduling by associating systemd units with specific metadata parameters. Metadata can be assigned either via cloud-config parameters passed when the instance is spawned, or via modifications to the fleet config files.

To see the units about which the cluster knows, use this command:

fleetctl list-units

If you’re just getting your etcd cluster up and running, the output of this command is probably empty. Let’s deploy a unit that spawns a Docker container running the popular Nginx web server. Here’s a (very) simple unit file that will spin up an Nginx container via Docker:

(If you can’t see the code block above, click here.)

With this file in place on the system where you are running fleetctl, you can submit this to the etcd cluster with this command:

fleetctl submit nginx.service

Then, when you run fleetctl list-units, you’ll see the new unit submitted (but not started). Start it with fleetctl start nginx.service.

Where fleet becomes really useful (in my opinion) is when you want to run multiple units across the cluster. If you take the simple Nginx unit I showed you earlier and extend it slightly, you get this:

(Click here if you can’t see the code block above.)

Note the difference here: the Docker container name is changed (to nginx-01) and the filename is different (now nginx.1.service). If you make multiple copies of this file, changing the Docker container name and the unit filename, you can submit all of the units to the etcd cluster at the same time. For example, let’s say you wanted to run 3 Nginx containers on the cluster. Make three copies of the file (nginx.1.service, nginx.2.service, and nginx.3.service), modifying the container name in each copy. Make sure that you have the “X-Conflicts” line in there; that tells fleet not to place two Nginx containers on the same system in the cluster. Then submit them with this command:

fleetctl submit nginx.*.service

And start (launch) them with this command:

fleetctl start nginx.*.service

Give it a few minutes to download the latest Nginx Docker image (assuming it isn’t already downloaded), then run fleetctl list-units and you should see three Nginx containers distributed across three different CoreOS instances in the etcd cluster, all listed as “loaded” and “active”. (You can then test connectivity to those Nginx instances using something like curl.) Congratulations—you’ve just deployed multiple containers automatically across a cluster of systems!

(Want to see some of the magic behind fleet? Run etcdctl --peers <IP address of cluster node>:4001 ls /_coreos.com --recursive and see what’s displayed. You’re welcome.)

Admittedly, this is a very simple example. However, the basic architecture I’ve shown you here can be extended. For example, by using additional fleet-specific properties like “X-ConditionMachineOf” in your unit file(s), you can run what is known as a “sidekick container.” These containers do things like update an external load balancer, or register the presence of the “primary” container in some sort of service discovery mechanism. (In fact, as I alluded to in my etcd post, you could use etcd as that service discovery mechanism.)

Naturally, fleetctl includes commands for stopping units, destroying units, etc., as well as submitting and starting units. You can use fleetctl help to get more information, or visit the fleet GitHub page.

I hope you’ve found this post to be helpful. Feel free to post any questions, corrections, clarifications, or thoughts in the comments below. Courteous comments are always welcome.

Tags: , , ,

CoreOS Continued: etcd

In this post, I’m going to build on my earlier introduction to CoreOS by taking a slightly more detailed look at etcd. etcd is a distributed key-value store (more on that in a moment) and is one of the key technologies that I feel distinguishes CoreOS from other Linux distributions.

etcd is not, of course, the only distributed key-value store in town. Zookeeper is another very well-known distributed information store. I’m focusing on etcd here for two reasons. First, it’s distributed as part of CoreOS, which means that it isn’t necessarily a separate piece of software that you have to install and configure (although some configuration is needed, as I’ll attempt to show). Second, future blog posts will talk about tools like fleet and others that are being built to leverage etcd, so I want to provide this overview here as a foundation for later posts.

Data Organization in etcd

As I’ve mentioned already, etcd is a distributed key-value store. The data within etcd is organized hierarchically, somewhat like a file system. The top of the hierarchy is “/” (root), and from there users can create paths and keys (with values). Later in this post, I’ll show you how that’s done using both the RESTful etcd API as well as using the etcdctl command-line client.

First, though, let’s look at an example. Let’s say that you wanted to store information about IP endpoints that were running a particular service. You could organize the data within etcd like this:

/ (root, already exists when etcd starts)
/svclocation (path or directory within etcd)
/svclocation/instance1 = 10.1.1.20 (a key in etcd and it’s associated value)
/svclocation/instance2 = 10.1.1.21 (another key and associated value)

Because etcd is a distributed key-value store, both the organization of the information as well as the actual information is distributed across all the members of an etcd cluster. If you’re interested in more details on exactly how the data is propagated within an etcd cluster, see here.

Configuration of etcd

etcd can be configured via command-line flags, environment variables, or configuration file. By default, etcd on CoreOS uses a configuration file generated by cloud-init to set environment variables that affect how etcd operates. My earlier post on deploying CoreOS on OpenStack with Heat provides a brief example of using cloud-init to configure etcd on CoreOS during deployment.

For example, when you configure the addr, peer-addr, or discovery properties via cloud-init, CoreOS puts those values into a file named 20-cloudinit.conf in the /run/systemd/system/etcd.service.d/ directory. That file takes those values and assigns them to the ETCD_ADDR, ETCD_PEER_ADDR, and ETCD_DISCOVERY environment variables, respectively. etcd will read those environment variables when it starts, thus controlling the configuration of etcd. (By the way, the use of the etcd.service.d directory for configuration files is a systemd thing.)

Interacting with etcd

Like most projects these days, etcd provides a RESTful API by which you can interact with it. You can use the command-line utility curl to interact with etcd, using some of the techniques I outlined in this post on interacting with RESTful APIs. Here’s a very simple example: using curl to recursively list the keys and values in a path within etcd:

curl -X GET http://10.1.1.7:4001/v2/keys/?consistent=true&recursive=true&sorted=false

If you want to store some data in etcd (let’s say you wanted to store the value “10.1.1.20:80″ at the key “/svclocation”), then you’d do something like this:

curl -L http://10.1.1.7:4001/v2/keys/svclocation -X PUT -d value="10.1.1.20:80"

The JSON response (read this if you’re unfamiliar with JSON) will provide confirmation that the key and value were set, along with some additional information.

To read this value back, you’d use curl like this:

curl -L http://10.1.1.7:4001/v2/keys/svclocation

etcd would respond with a JSON-formatted response that provides the current value of the specified key.

However, to make it easier, there is a command-line client called etcdctl that you can use to interact with etcd. There are both Linux and OS X versions available from the etcd GitHub page; just make sure you download the version that corresponds to the version of etcd that’s running on your CoreOS instance(s). (To determine the version of etcd running on a CoreOS instance, log into the CoreOS instance via SSH and run etcd --version.)

Then, to perform the same listing of keys action as the curl command I provided earlier, you would run this command:

etcdctl --peers 10.1.1.7:4001 ls / --recursive

Similarly, to set a value at a particular key within etcd:

etcdctl --peers 10.1.1.7:4001 mk /svclocation 10.1.1.20:80

And to get a value:

etcdctl --peers 10.1.1.7:4001 get /svclocation

A couple of points to note:

  • By default, etcdctl works against a local instance of etcd. In other words, it defaults to connecting to 127.0.0.1:4001. You’ll have to use the --peers flag if you’re running it external to a CoreOS instance. However, if you’re running it directly from a CoreOS instance, you can omit the --peers flag.
  • You can point etcdctl against any CoreOS instance in an etcd cluster, because the information stored in etcd is shared and distributed across all the members of the cluster. (That’s kind of the whole point behind it.) Note that the --peers option supports listing multiple peers in an etcd cluster.

By now, you might be thinking, “OK, this is interesting, but what does it really do for me?” By itself…not necessarily a whole lot. Where etcd starts to become quite useful is when there are other systems that start to leverage information stored in etcd. A couple of examples come to mind:

  1. Distributed micro-service architectures—where applications are made up of a group of containerized services distributed across a compute farm—need mechanisms for service registration (registering that a particular service is available) and service discovery (finding out where a particular service is running). etcd could be helpful in assisting with both of these tasks (writing key-value pairs into etcd for registration, reading them back for discovery).

  2. “Converting” etcd information into traditional configuration files allows applications that are not etcd-aware to still take advantage of etcd’s distributed architecture. For example, there is a project called confd that takes information stored in etcd and turns it into “standard” configuration files. This means, for example, that you could store dynamic configuration details in etcd, and then use confd to update static configuration files. (A common example of using this is updating an HAProxy configuration file via confd as back-end web server containers start up and shut down. Astute readers will recognize this as a form of service registration and service discovery.)

I’ll be building on some of these concepts in future posts. In the meantime, feel free to post any questions, clarifications, or corrections in the comments below.

Tags: , , ,

In this post, I’m going to illustrate one way to deploy CoreOS on OpenStack using Heat. By no means is this intended to be seen as the only way to use Heat to deploy CoreOS, but rather as one way of using Heat to deploy CoreOS. I’m publishing this in the hopes that others will be able to use this as a building block for their own deployments.

If you aren’t already familiar with OpenStack Heat or CoreOS, you might want to take a moment and refer to this introductory posts for some foundational information:

Moving forward, OpenStack Heat is trying to standardize on OpenStack resource types (like OS::Nova::Server) and the HOT format (using YAML). Therefore, the Heat template I’m presenting here will use OpenStack resource types and YAML. Note that it’s certainly possible to do this using CloudFormation (CFN) resource types and JSON formatting. I’ll leave the conversion of the template found here into CFN/JSON as an exercise for the readers.

Here’s the example Heat template you can use to deploy and customize CoreOS on OpenStack:

(Click here if you can’t see the code block above.)

Let’s walk through this template real quick:

  • On line 9, you’ll need to provide the ID for the Neutron network to which the new CoreOS instance(s) should connect. You can get this a couple of different ways; running neutron net-list is one way.
  • On line 14, you’ll need to supply the ID for the CoreOS image you’ve uploaded into Glance. Again, there are multiple ways to obtain this; running glance image-list is one way of getting that information.
  • On line 22, replace the text (including the “<” and “>” symbols) with the ID of the security group you want applied to the CoreOS instance(s) being deployed. The neutron security-group-list command can give you the information you need to put here.
  • On line 31, supply the name of the SSH key you want to inject into the instance(s).
  • On line 37, you’ll need to generate a unique cluster ID to place here for the configuration of etcd within the CoreOS instance(s). You can generate a new ID (also called a token) by visiting https://discovery.etcd.io/new. That will return another URL that contains the new etcd cluster token. Supply that token here to create a new etcd cluster out of the CoreOS instance(s) you’re deploying with this template.
  • This template only deploys a single CoreOS instance. To deploy multiple CoreOS instances, you’ll need a separate OS::Neutron::Port and OS::Nova::Server resource for each instance. For each Neutron port, you can reference the same security group ID and network ID. For each instance, you can reference the same Glance image ID, same SSH key, and same etcd cluster token; the only thing that would change with each instance is line 30. Line 30 should point to a unique Neutron port resource created for each instance (something like instance1_port0, instance2_port0, etc.).

Now, there are obviously lots of other things you could do here—you could create your own Neutron network to host these CoreOS instances, you could create a logical router to provide external connectivity (which is required, by the way, in order for the etcd cluster token discovery to work correctly), and you could create and assign floating IPs to the instances. Examples of some of these tasks are in the articles I provided earlier; others are left as an exercise for the reader. (Or I’ll write up something later. We’ll see.)

Once you have your template, you can deploy the stack using Heat, and then—after your CoreOS cluster is up and running—begin to deploy applications to the cluster using tools like fleet. That, my friends, is another story for another day.

Any questions? Corrections? Clarifications? Feel free to start (or join) the discussion below. All courteous comments are welcome.

Tags: , , ,

Some time ago, I introduced you to the idea of Linux network namespaces, and provided an overview of some of the commands needed to interact with network namespaces. In this post, I’ll follow up on that post with some additional information on using network namespaces with other types of network interfaces.

In the previous network namespaces article, I mentioned (incorrectly, it turns out) that you had to use virtual Ethernet (veth) interfaces in order to connect a namespace to the physical network:

It turns out you can only assign virtual Ethernet (veth) interfaces to a network namespace.

I say “incorrectly” because you are able to assign more than just virtual Ethernet interfaces to a Linux network namespace. I’m not sure why I arrived at that conclusion, because subsequent testing—using Ubuntu 12.04 LTS as with the original testing—showed no problem assigning physical interfaces to a particular network namespace. The VMs I used for the subsequent testing are using a different kernel (the 3.11 kernel instead of whatever I used in the previous testing), so it’s entirely possible that’s the difference. If I get the opportunity, I’ll try with earlier kernel builds to see if that makes any difference.

In any case, assigning other types of network interfaces to a network namespace is just like assigning veth interfaces. First, you create the network namespace:

ip netns add <new namespace name>

Then, you’d assign the interface to the namespace:

ip link set <device name> netns <namespace name>

For example, if you wanted to assign eth1 to the “blue” namespace, you’d run this:

ip link set eth1 netns blue

Please note that I haven’t found a way to unassign an interface from a network namespace other than deleting the namespace entirely.

If you want to assign a VLAN interface to a namespace, the process is slightly different. You’ll have to create the namespace first, as with physical and veth interfaces, but you’ll also have to create the VLAN interface in the “default” namespace first, then move it over to the desired namespace.

For example, first you’d create the namespace “red”:

ip netns add red

Then you’d create the VLAN interface for VLAN 100 on physical interface eth1:

ip link add link eth1 name eth1.100 type vlan id 100

The generic form of this command is this:

ip link add link <physical device> name <VLAN device name> type vlan id <VLAN ID>

Note that you can’t use ip netns exec to run the command to create the VLAN interface in the network namespace directly; it won’t work because the parent interface upon which the VLAN interface is based doesn’t exist in the namespace. So you’ll create the VLAN interface in the default namespace first, then move it over.

Once the VLAN interface is created, then move it to the target namespace:

ip link set eth1.100 netns red

One (sort of) interesting thing I noted in my testing was that link status and IP addresses don’t move between namespaces. Therefore, don’t bother assigning an IP address or setting the link state of an interface before you move it to the final namespace, because you’ll just have to do it again.

To make the interface functional inside the target namespace, you’ll use ip netns exec to target the specific configuration commands against the desired namespace. For example, if the VLAN interface eth1.100 exists in the namespace “blue”, you’d run these commands:

ip netns exec blue ip addr add 10.1.1.1/24 dev eth1.100

That adds the IP address to the interface in the namespace; then you’d use this command to set the link status to up:

ip netns exec blue ip link set eth1.100 up

Then you can test network connectivity with our good friend ping like this:

ip netns exec blue ping -c 4 10.1.1.2

(Obviously, you’d want to substitute an appropriate IP address there for your specific configuration and environment.)

I hope this additional information on working with Linux network namespaces is useful. As always, I invite and encourage any questions, thoughts, or corrections in the comments below. All courteous comments are welcome!

Tags: , ,

In this post, I’m going to talk about a way to modify QCOW2-based images without having to boot up a full-blown virtual machine based on the image. I found this process while trying to change the behavior of cloud-init on the official Ubuntu cloud images from Canonical.

While I had successfully built my own Ubuntu 12.04 LTS image for my OpenStack-based home lab (and it worked just fine), I was keen to use Canonical’s official cloud images as they were much smaller (around 250MB versus 1.5GB for my home-built image). Given that disk space is at a bit of a premium in my home lab, using the smaller images would be quite beneficial. The problem was that cloud-init wasn’t configured to operate in a way that worked for my home lab environment.

The question, of course, was…how do I do this? I tried a couple different approaches:

  1. I installed VirtualBox on one of my OS X systems, because I knew VirtualBox supported QCOW2 images. Unfortunately, trying to launch a VM using one of the official cloud images as a root disk failed. (I presume this is because the cloud images assume the use of the virtio block driver, which VirtualBox doesn’t support.)

  2. Next, I installed a plain jane Ubuntu VM in VirtualBox and attached the official cloud image QCOW2 file as a second hard disk. That didn’t work either; the Ubuntu VM refused to recognize the filesystem(s) on the QCOW2 image.

I was about at my wits’ end when I found this post. Aha! I’d heard of guestfish before, but I hadn’t thought to try it in this instance. Guestfish is part of the larger libguestfs project.

So I tried the instructions in the post I’d found. Unfortunately, it failed the first time I tried, but I later—thanks to qemu-img check—found that the Ubuntu 12.04 cloud image I’d downloaded was somehow damaged/corrupted. Using a new download of the image was successful.

Here are the steps I used to modify the contents of a QCOW2 disk image using guestfish:

  1. Starting from a vanilla Ubuntu 12.04 LTS install, use apt-get update and apt-get install guestfish to install guestfish. (You’ll probably need/want to use sudo with both of these commands.)

  2. During the installation of guestfish, you can select to create a “virtual appliance” that guestfish uses. If you choose not to do this during installation, you can always do it later with sudo update-guestfs-appliance.

  3. Once guestfish is installed and the virtual appliance is created/updated, launch guestfish itself with sudo guestfish. Note that if you don’t have root privileges or don’t use sudo, guestfish will still launch and act like everything is working—but it will fail. Once guestfish is up and running, you’ll be sitting at a “> <fs>” prompt.

  4. Type add <QCOW2 image filename> (obviously substituting the full path and correct filename for the QCOW2 image you’d like to modify).

  5. Type run and press Enter. You’ll see a progress bar that updates the status of the guestfs virtual appliance. Once it’s complete, you’ll be returned to the “> <fs>” prompt.

  6. Enter list-filesystems. Guestfish should return a list of filesystems on the QCOW2 disk image. In the case of the official Canonical cloud images, only “/dev/vda1″ is returned.

  7. To gain access to the filesystem in the QCOW2 image, enter mount <device specification> <guestfish mount point>. For example, when I was modifying the Canonical cloud images, I used mount /dev/vda1 /. Keep in mind you’re not mounting to the system on which guestfish is running; you’re mounting to the guestfish virtual appliance.

  8. Now just edit the files in the QCOW2 image using whatever method you prefer. In my case, I needed to edit /etc/cloud/cloud.cfg, so I just used vi to edit the file and make whatever changes were needed.

  9. When you’re done, type exit to quit guestfish and commit changes to the QCOW2 image.

And that’s it! Using this technique I was able to quickly and easily modify the Ubuntu cloud images so that cloud-init behaved in the way that I wanted.

Hopefully this information helps someone else. If you have any questions or corrections, I invite you to share in the comments below.

Tags: , , ,

It seems as if APIs are popping up everywhere these days. While this isn’t a bad thing, it does mean that IT professionals need to have a better understanding of how to interact with these APIs. In this post, I’m going to discuss how to use the popular command line utility curl to interact with a couple of RESTful APIs—specifically, the OpenStack APIs and the VMware NSX API.

Before I go any further, I want to note that to work with the OpenStack and VMware NSX APIs you’ll be sending and receiving information in JSON (JavaScript Object Notation). If you aren’t familiar with JSON, don’t worry—I’ve have an introductory post on JSON that will help get you up to speed. (Mac users might also find this post helpful as well.)

Also, please note that this post is not intended to be a comprehensive reference to the (quite extensive) flexibility of curl. My purpose here is to provide enough of a basic reference to get you started. The rest is up to you!

To make consuming this information easier (I hope), I’ll break this information down into a series of examples. Let’s start with passing some JSON data to a REST API to authenticate.

Example 1: Authenticating to OpenStack

Let’s say you’re working with an OpenStack-based cloud, and you need to authenticate to OpenStack using OpenStack Identity (“Keystone”). Keystone uses the idea of tokens, and to obtain a token you have to pass correct credentials. Here’s how you would perform that task using curl.

You’re going to use a couple of different command-line options:

  • The “-d” option allows us to pass data to the remote server (in this example, the remote server running OpenStack Identity). We can either embed the data in the command or pass the data using a file; I’ll show you both variations.
  • The “-H” option allows you to add an HTTP header to the request.

If you want to embed the authentication credentials into the command line, then your command would look something like this:

curl -d '{"auth":{"passwordCredentials":{"username": "admin",
"password": "secret"},"tenantName": "customer-A"}}'
-H "Content-Type: application/json" http://192.168.100.100:5000/v2.0/tokens

I’ve wrapped the text above for readability, but on the actual command line it would all run together with no breaks. (So don’t try to copy and paste, it probably won’t work.) You’ll naturally want to substitute the correct values for the username, password, tenant, and OpenStack Identity URL.

As you might have surmised by the use of the “-H” header in that command, the authentication data you’re passing via the “-d” parameter is actually JSON. (Run it through python -m json.tool and see.) Because it’s actually JSON, you could just as easily put this information into a file and pass it to the server that way. Let’s say you put this information (which you could format for easier readability) into a file named credentials.json. Then the command would look something like this (you might need to include the full path to the file):

curl -d @credentials.json -H "Content-Type: application/json" http://192.168.100.100:35357/v2.0/tokens

What you’ll get back from OpenStack—assuming your command is successful—is a wealth of JSON. I highly recommend piping the output through python -m json.tool as it can be difficult to read otherwise. (Alternately, you could pipe the output into a file.) Of particular usefulness in the returned JSON is a section that gives you a token ID. Using this token ID, you can prove that you’ve authenticated to OpenStack, which allows you to run subsequent commands (like listing tenants, users, etc.).

Example 2: Authenticating to VMware NSX

Not all RESTful APIs handle authentication in the same way. In the previous example, I showed you how to pass some credentials in JSON-encoded format to authenticate. However, some systems use other methods for authentication. VMware NSX is one example.

In this example, you’ll need to use a different set of curl command-line options:

  • The “–insecure” option tells curl to ignore HTTPS certificate validation. VMware NSX controllers only listen on HTTPS (not HTTP).
  • The “-c” option stores data received by the server (one of the NSX controllers, in this case) into a cookie file. You’ll then re-use this data in subsequent commands with the “-b” option.
  • The “-X” option allows you to specify the HTTP method, which normally defaults to GET. In this case, you’ll use the POST method along the the “-d” parameter you saw earlier to pass authentication data to the NSX controller.

Putting all this together, the command to authenticate to VMware NSX would look something like this (naturally you’d want to substitute the correct username and password where applicable):

curl --insecure -c cookies.txt -X POST -d 'username=admin&password=admin' https://192.168.100.50/ws.v1/login

Example 3: Gathering Information from OpenStack

Once you’ve gotten an authentication token from OpenStack as I showed you in example #1 above, then you can start using API requests to get information from OpenStack.

For example, let’s say you wanted to list the instances for a particular tenant. Once you’ve authenticated, you’d want to get the ID for the tenant in question, so you’d need to ask OpenStack to give you a list of the tenants (you’ll only see the tenants your credentials permit). The command to do that would look something like this:

curl -H "X-Auth-Token: <Token ID>" http://192.168.100.70:5000/v2.0/tenants

The value to be substituted for token ID in the above command is returned by OpenStack when you authenticate (that’s why it’s important to pay attention to the data being returned). In this case, the data returned by the command will be a JSON-encoded list of tenants, tenant IDs, and tenant descriptions. From that data, you can get the ID of the tenant for whom you’d like to list the instances, then use a command like this:

curl -H "X-Auth-Token: <Token ID>" http://192.168.100.70:8774/v2/<Tenant ID>/servers

This will return a stream of JSON-encoded data that includes the list of instances and each instance’s unique ID—which you could then use to get more detailed information about that instance:

curl -H "X-Auth-Token: <Token ID>" http://192.168.100.70:8774/v2/<Tenant ID>/servers/<Server ID>

By and large, the API is reasonably well-documented; you just need to be sure that you are pointing the API call against the right endpoint. For example, authentication has to happen against the server running Keystone, which may or may not be the same server that is running the Nova API services. (In the examples I just provided, Keystone and the Nova API services are running on the same host, which is why the IP address is the same in the command lines.)

Example 4: Creating Objects in VMware NSX

Getting information from VMware NSX using the RESTful API is very much like what you’ve already seen in getting information from OpenStack. Of course, the API can also be used to create objects. To create objects—such as logical switches, logical switch ports, or ACLs—you’ll use a combination of curl options:

  • You’ll use the “-b” option to pass cookie data (stored when you authenticated to NSX) back for authentication.
  • The “-X” option allows you to specify the HTTP method (in this case, POST).
  • The “-d” option lets us transfer JSON-encoded data to form the request for the object we are going to create. We’ll specify a filename here, preceded by the “@” symbol.
  • The “-H” option adds an appropriate “Content-Type: application/json” header to the request, since we are passing JSON-encoded data to the NSX controller.

When you put it all together, it looks something like this (substituting appropriate values where applicable):

curl --insecure -b cookies.txt -d @new-switch.json 
-H "Content-Type: application/json" -X POST https://192.168.100.50/ws.v1/lswitch

As I mentioned earlier, you’re passing JSON-encoded data to the NSX controller; here are the contents of the new-switch.json file referenced in the above command example:

If you can’t see the code block, please click here.

Once again, I recommend piping the output of this command through python -m json.tool, as what you’ll get back on a successful call is some useful JSON data that includes, among other things, the UUID of the object (logical switch, in this case) that you just created. You can use this UUID in subsequent API calls to list properties, change properties, add logical switch ports, etc.

Clearly, there is much more that can be done with the OpenStack and VMware NSX APIs, but this at least should give you a starting point from which you can continue to explore in more detail. If anyone has any corrections, clarifications, or questions, please feel free to post them in the comments section below. All courteous comments (with vendor disclosure, where applicable) are welcome!

Tags: , , , ,

In this article, I’ll show you how to install Ubuntu packages from a specific repository. It’s not that this is a terribly difficult process, but it also isn’t necessarily intuitive for those who haven’t had to do it before. I ran into this while trying to install an alpha release of LXC 1.0.0 for my recent post on automatically connecting LXC to Open vSwitch (OVS).

Normally, you’d install a package using apt-get like this:

apt-get install package-name

However, when you want to install a package from a specific repository, the command syntax shifts slightly so that it looks like this instead:

apt-get install package-name/repository

So, in my particular case, I was trying to install LXC. However, I needed the alpha LXC 1.0.0 package from the precise-backports repository. So the command to do that looked like this:

apt-get install lxc/precise-backports

Are there other useful apt-get tidbits like this that other readers might find particularly useful? Feel free to share in the comments below. Thanks for reading!

Tags: , ,

« Older entries