You are currently browsing articles tagged Ubuntu.

In this article, I’ll show you how to install Ubuntu packages from a specific repository. It’s not that this is a terribly difficult process, but it also isn’t necessarily intuitive for those who haven’t had to do it before. I ran into this while trying to install an alpha release of LXC 1.0.0 for my recent post on automatically connecting LXC to Open vSwitch (OVS).

Normally, you’d install a package using apt-get like this:

apt-get install package-name

However, when you want to install a package from a specific repository, the command syntax shifts slightly so that it looks like this instead:

apt-get install package-name/repository

So, in my particular case, I was trying to install LXC. However, I needed the alpha LXC 1.0.0 package from the precise-backports repository. So the command to do that looked like this:

apt-get install lxc/precise-backports

Are there other useful apt-get tidbits like this that other readers might find particularly useful? Feel free to share in the comments below. Thanks for reading!

Tags: , ,

I’ve previously discussed using Open vSwitch (OVS) with Linux Containers (LXC) in a couple of previous posts (here and here). In this post, I’m going to show you one way to have your containers automatically connected to OVS on startup without having to use libvirt.

I tested this configuration using Ubuntu 12.04 with the Linux 3.8.0 kernel and an alpha release of LXC 1.0.0 from the precise-backports repository. The version of LXC in the 12.04 repositories (version 0.7.5, if I recall correctly) isn’t new enough to support the specific feature I’m describing here, so plan accordingly.

If you aren’t familiar with LXC, I’d suggest you first read my LXC introductory post. You’ll probably also find the post on using LXC, OVS, and GRE tunnels useful, as some of the information there is applicable here also.

Ready? Let’s get started.

Configuring the Container

You can create your container using the standard LXC tools:

lxc-create -n cn-01 -t ubuntu

As you may already know, this will create a container named “cn–01″ based on the Ubuntu template. The configuration for this container will be found, by default, at /var/lib/lxc/cn–01/config. By default, unless you’ve changed the configuration of your system, this container will be configured to use virtual Ethernet network interfaces and be attached to the default LXC bridge.

The changes required to make the container connect to OVS are, fortunately, quite minimal:

  1. First, remove or comment out the lxc.network.link line. This is the configuration parameter that causes the container to attach to the default LXC bridge (normally called “lxcbr0″).
  2. Add a configuration line to run a script after creating the network interfaces. In my examples here, I’ll assume the script is called “ovsup” and is stored in the /etc/lxc/ directory. The configuration parameter should look something like this:
lxc.network.script.up = /etc/lxc/ovsup

(Note that there is also a corresponding lxc.network.script.down configuration parameter, but I won’t be using it in this example.)

Once you’ve made these changes to the container’s configuration, then you’re ready to create the actual script.

Creating the Network Attachment Script

Your script—the one referenced on the lxc.network.script.up in the container’s configuration file—should look something like this:

(If you can’t see the code block above, please click here.)

LXC passes five parameters to the script when it is called:

  1. The name of the container
  2. The configuration section of the container’s configuration (“net” in this case)
  3. Either “up” or “down”, depending on which configuration option is calling the script (lxc.network.script.up passes “up”, lxc.network.script.down passes “down”)
  4. The type of networking (“veth” in this case)
  5. The name of the interface (randomly generated unless you have included lxc.network.veth.peer in the container’s configuration)

This simple script doesn’t really need anything other than the interface name, so it only uses parameter 5 (the $5 in the script). The script first ensures that the appropriate OVS bridge exists (creating it if necessary), then deletes the interface from the OVS bridge (if it exists) and adds it back to the OVS bridge.

(Note: If you are using the lxc.network.script.down configuration parameter, you could eliminate the line to delete the port from the OVS bridge and place it in the down script instead. Or, you could write logic into the script to see if “down” is being called and delete the port. There are a variety of ways to approach the situation.)

Using this configuration, when you start the container the host-side virtual Ethernet interface created by LXC will be automatically added to OVS, and your container will have whatever network connectivity is dictated by the OVS configuration. This could include tunneled connectivity (as described here) or bridged connectivity.

If you have any questions, feedback, or corrections, please feel free to speak up in the comments below. I encourage reader interaction!

Tags: , , ,

For the last couple of years, I’ve been sharing my annual “projects list” and then grading myself on the progress (or lack thereof) on the projects at the end of the year. For example, I shared my 2012 project list in early January 2012, then gave myself grades on my progress in early January 2013.

In this post, I’m going to grade myself on my 2013 project list. Here’s the project list I posted just under a year ago:

  1. Continue to learn German.
  2. Reinforce base Linux knowledge.
  3. Continue using Puppet for automation.
  4. Reinforce data center networking fundamentals.

So, how did I do? Here’s my assessment of my progress:

  1. Continue to learn German: I have made some progress here, though certainly not the progress that I wanted to learn. I’ve incorporated the use of Memrise, which has been helpful, but I still haven’t made the progress I’d like. If anyone has any other suggestions for additional tools, I’m open to your feedback. Grade: D (below average)

  2. Reinforce base Linux knowledge: I’ve been suggesting to VMUG attendees that they needed to learn Linux, as it’s popping up all over the place in all sorts of roles. In my original 2013 project list, I said that I was going to focus on RHEL and RHEL variants, but over the course of the year ended up focusing more on Debian and Ubuntu instead (due to more up-to-date packages and closer alignment with OpenStack). Despite that shift in focus, I think I’ve made decent progress here. There’s always room to grow, of course. Grade: B (above average)

  3. Continue using Puppet for automation: I’ve made reasonable progress here, expanding my use of Puppet to include managing Debian/Ubuntu software repositories (see here and here for examples), managing SSH keys, managing Open vSwitch (OVS) via a third-party module, and—most recently—exploring the use of Puppet with OpenStack (no blog posts—yet). There’s still quite a bit I need to learn (some of my manifests don’t work quite as well as I’d like), but I did make progress here. Grade: C (average)

  4. Reinforce data center networking fundamentals: Naturally, my role at VMware has me spending a great deal of time on how network virtualization affects DC networking, and this translated into some progress on this project. While I gained solid high-level knowledge on a number of DC networking topics, I think I was originally thinking I needed more low-level “in the weeds” knowledge. In that regard, I don’t feel like I did well; on the flip side, though, I’m not sure whether I really needed more low-level “in the weeds” knowledge. This highlights a key struggle for me personally: how to balance the deep, “in the weeds” knowledge with the high-level knowledge. Suggestions on how others have overcome this challenge are welcome. Grade: C (average)

In summary: not bad, but could have been better!

What’s not reflected in this project list is the progress I made with understanding OpenStack, or my deepened level of knowledge of OVS (just browse articles tagged OVS for an idea of what I’ve been doing in that area).

Over the next week or two, I’ll be reflecting on my progress with my 2013 projects and thinking about what projects I should be taking in 2014. In the meantime, I would love to hear any feedback, suggestions, or thoughts on projects I should consider, technologies that should be incorporated, or learning techniques I should leverage. Feel free to speak up in the comments below.

Tags: , , , , , , ,

Some time ago, I showed you how to use Puppet to add Ubuntu Cloud Archive support to your Ubuntu installation. Since that time, OpenStack has had a new release (the Havana release) and the Ubuntu Cloud Archive repository has been updated with new packages to support the Havana release. In this post, I’ll show you an updated snippet of code to take advantage of these newer packages in the Ubuntu Cloud Archive repository.

For reference, here’s the original Puppet code I posted in the first article:

(If you can’t see the code snippet above, please click here.)

That points your Ubuntu installation to the Grizzly packages.

Here’s updated code that will point your installation to the appropriate packages to support OpenStack’s Havana release:

(Click here if you can’t see the code snippet above.)

As you can see, there is only one small change between the two code snippets: changing “precise-updates/grizzly” in the first to “precise-updates/havana” in the second. (Naturally, this assumes you’re using Ubuntu 12.04, the latest LTS release as of this writing.) I know this seems like a pretty simple thing to post, but I wanted to include it here for the sake of completeness and the benefit of future readers.

Feel free to speak up in the comments with any questions, suggestions, or corrections.

Tags: , , , ,

I’ve been doing some experimenting with virtual Ethernet (veth) interfaces in Ubuntu as part of the ongoing work with network namespaces, LXC, and related technologies. A few times I’ve run into a very weird situation, and I have yet to figure out exactly what’s happening. I thought I might share it here in the hopes that someone else has seen this behavior and knows a) what causes it, and b) how to fix it.

I’ll start with a pretty vanilla installation of Ubuntu 12.04 LTS and Open vSwitch (OVS). When I run ip link list, I get output that looks something like this (click the image for a larger version):

Before adding the veth pair

OK, nothing unusual or unexpected there.

Next, I’ll add a pair of veth interfaces:

ip link add vmveth0 type veth peer vmveth1

Then the output of ip link list looks like this (I’ve circled some of the output to draw your attention; again, you can click for a larger version):

After adding the veth pair

See? The name of the veth peer interface gets garbled up and somehow corrupted. Because of this, nothing works—I can’t use the veth pair to connect network namespaces, or to connect a Linux bridge to OVS, or anything else. Rebooting the system does not fix the problem; only a rebuild seems to get rid of it.

Anyone have any ideas?

Tags: , , ,

In this post, I’m going to provide a brief introduction to working with Linux containers via LXC. Linux containers are getting a fair amount of attention these days (perhaps due to Docker, which leverages LXC on the back-end) as a lightweight alternative to full machine virtualization such as that provided by “traditional” hypervisors like KVM, Xen, or ESXi.

Both full machine virtualization and containers have their advantages and disadvantages. Full machine virtualization offers greater isolation at the cost of greater overhead, as each virtual machine runs its own full kernel and operating system instance. Containers, on the other hand, generally offer less isolation but lower overhead through sharing certain portions of the host kernel and operating system instance. In my opinion full machine virtualization and containers are complementary; each offers certain advantages that might be useful in specific situations.

Now that you have a rough idea of what containers are, let’s take a closer look at using containers with LXC. I’m using Ubuntu 12.04.3 LTS for my testing; if you’re using something different, keep in mind that certain commands may differ from what I show you here.

Installing LXC is pretty straightforward, at least on Ubuntu. To install LXC, simply use apt-get:

apt-get install lxc

Once you have LXC installed, your next step is creating a container. To create a container, you’ll use the lxc-create command and supply the name of the container template as well as the name you want to assign to the new container:

lxc-create -t <template> -n <container name>

You’ll need Internet access to run this command, as it will download (via your configured repositories) the necessary files to build a container according to the template you specified on the command line. For example, to use the “ubuntu” template and create a new container called “cn–01″, the command would look like this:

lxc-create -t ubuntu -n cn-01

Note that the “ubuntu” template specified in this command has some additional options supported; for example, you can opt to create a container with a different release of Ubuntu (it defaults to the latest LTS) or a different architecture (it defaults to the host’s architecture).

Once you have at least one container created, you can list the containers that exist on your host system:


This will show you all the containers that have been created, grouped according to whether the container is stopped, frozen (paused), or running.

To start a container, use the lxc-start command:

lxc-start -n <container name>

Using the lxc-start command as shown above is fine for initial testing of your container, to ensure that it boots up as you expect. However, you won’t want to run your containers long-term like this, as the container “takes over” your console with this command. Instead, you want the container to run in the background, detached from the console. To do that, you’ll add the “-d” parameter to the command:

lxc-start -d -n <container name>

This launches your container in the background. To attach to the console of the container, you can use the lxc-console command:

lxc-console -n <container name>

To escape out of the container’s console back to the host’s console, use the “Ctrl-a q” key sequence (press Ctrl-a, release, then press q).

You can freeze (pause) a container using the lxc-freeze command:

lxc-freeze -n <container name>

Once frozen, you can unfreeze (resume) a container just as easily with the lxc-unfreeze command:

lxc-unfreeze -n <container name>

You can also make a clone (a copy) of a container:

lxc-clone -o <existing container> -n <new container name>

On Ubuntu, LXC is configured by default to start containers in /var/lib/lxc. Each container will have a directory there. In a container’s directory, that container’s configuration will be stored in a file named config. I’m not going to provide a comprehensive breakdown of the settings available in the container’s configuration (this is a brief introduction), but I will call out a few that are worth noting in my opinion:

  • The lxc.network.type option controls what kind of networking the container will use. The default is “veth”; this uses virtual Ethernet pairs. (If you aren’t familiar with veth pairs, see my post on Linux network namespaces.)
  • The lxc.network.veth.pair configuration option controls the name of the veth interface created in the host. By default, a container sees one side of the veth pair as eth0 (as would be expected), and the host sees the other side as either a random name (default) or whatever you specify here. Personally, I find it useful to rename the host interface so that it’s easier to tell which veth interface goes to which container, but YMMV.
  • lxc.network.link specifies a bridge to which the host side of the veth pair should be attached. If you leave this blank, the host veth interface is unattached.
  • The configuration option lxc.rootfs specifies where the container’s root file system is stored. By default it is /var/lib/lxc/<container name>/rootfs.

There are a great deal of other configuration options, naturally; check out man 5 lxc.conf for more information. You may also find this Ubuntu page on LXC to be helpful; I certainly did.

I’ll have more posts on Linux containers in the future, but this should suffice to at least help you get started. If you have any questions, any suggestions for additional resources other readers should consider, or any feedback on the post, please add your comment below. I’d love to hear from you (courteous comments are always welcome).

Tags: , , , ,

In this post, I’ll share with you some Puppet code that you can include in your manifests to install Open vSwitch (OVS) packages on Ubuntu. This post, along with a number of others (like using Puppet for Ubuntu Cloud Archive support or using Puppet to configure an Apt proxy) stems from my work on building a new home lab in which I’ll be doing some OpenStack and NSX testing.

This code makes a couple of assumptions:

  1. It assumes that you’ve established an internal Apt repository (I created one using reprepro). In the code below, you’ll see that I’ve used the Puppet Labs Apt module to define my internal Apt repository.
  2. It assumes that you have Debian packages for OVS in that internal Apt repository. Depending on which version of OVS you need (I needed a newer version than was available in the public repositories), you might be able to get away with just using the public repositories.

OK, with the assumptions out of the way, let’s have a look at the code:

(Click here if the code block above isn’t visible.)

The code is fairly straightforward; the key is making sure that the appropriate packages are installed before you attempt to install the OVS DKMS module. This is reflected in the require statement for the openvswitch-datapath-dkms package.

I’ve only tested this on Ubuntu 12.04 LTS, so use at your own risk on other distributions and other versions.

As always, I encourage you to participate in the discussion by adding your questions, thoughts, suggestions, and/or clarifications in the comments below. All courteous comments are welcome.

Tags: , , , , ,

In this post, I’ll share a brief snippet of Puppet code that allows you to automatically configure Ubuntu clients to use Apt-Cacher-NG. By leveraging Apt-Cacher-NG, running apt-get commands on your Ubuntu instances will generally be faster because the Apt-Cacher-NG server will cache information locally instead of requiring that every command go out to the source repositories. In my own lab I’ve seen a tremendous speed boost on installing updates and frequently-used packages on my Ubuntu instances. You can get more information on Apt-Cacher-NG on the Apt-Cacher-NG website.

(Note: The Puppet code in this post relies upon the same Puppet Labs apt module that I used in my post on using Puppet to configure Ubuntu to use the Ubuntu Cloud Archive.)

This snippet of Puppet code will take care of configuring apt to use a local Apt-Cacher-NG instance:

(In the event the code block above isn’t shown, you can also see it here.)

This is a really simple block of code, but I’m publishing it here just for the sake of completeness and in the remote event someone else will find it useful. Because this a distro-specific thing (only applies to Debian and Debian derivatives like Ubuntu), you might want to wrap this in a conditional (like If $::osfamily == ‘Debian' or similar) to prevent errors in the event this manifest is (accidentally) applied to a non-Debian distribution.

Questions, corrections, and other feedback are welcome, so feel free to speak up in the comments below.

Tags: , , ,

In this post, I’ll share a snippet of Puppet code that I am using to automatically configure Ubuntu Server 12.04 systems to use the Ubuntu Cloud Archive (which allows access to packaged versions of OpenStack for use with LTS releases of Ubuntu Server, like 12.04).

As you may already know, I recently acquired two off-lease Dell PowerEdge C6100 systems. Each of these units has four trays; each tray is a dual-socket, quad-core server with 24GB of RAM. This gives me a total of eight servers, and the plan is to use them to build an internal OpenStack cloud running Ubuntu 12.04, KVM, Open vSwitch (OVS), and—ultimately—VMware NSX. It’s a fairly ambitious goal, but if you don’t stretch yourself you’ll never grow.

In any case, along the way I’m trying to make the whole process as repeatable and automated as possible, and naturally that’s where Puppet comes into play. I’ve been working through an automated Ubuntu Server install via PXE and an internal HTTP repository (I’ll do a separate post for that), but as part of my testing I wanted to be sure that I could automatically configure the Ubuntu Server instances to use the Ubuntu Cloud Archive for access to OpenStack packages. While this isn’t necessarily hard, I did want to share the Puppet code I’m using just in case it might help someone else.

First off, you’ll want to get your hands on the Puppet Labs apt module from the Forge. Once you’ve gotten that installed on your Puppet server (a simple puppet module install puppetlabs/apt on any recent version of Puppet should knock that out for you), then you can use this snippet of code in a manifest:

(In case the code above doesn’t show up, you can also view it here.)

Once you put this into the Puppet manifest and then refresh the system’s configuration, you should see a file named ubuntu-cloud.list appear in the /etc/apt/sources.list.d directory on your Ubuntu system. (By the way, I usually wrap that code in a conditional like if $::operatingsystem == ‘Ubuntu' or similar.) Once that file is there, simply run apt-get update and you should now be able to install packages from the Ubuntu Cloud Archive.

Have fun!

Tags: , , , ,

In this post, I’m going to show you how I combined Linux network namespaces, VLANs, Open vSwitch (OVS), and GRE tunnels to do something interesting. Well, I found it interesting, even if no one else does. However, I will provide this disclaimer up front: while I think this is technically interesting, I don’t think it has any real, practical value in a production environment. (I’m happy to be proven wrong, BTW.)

This post builds on information I’ve provided in previous posts:

It may pull pieces from a few other posts, but the bulk of the info is found in these. If you haven’t already read these, you might want to take a few minutes and go do that—it will probably help make this post a bit more digestible.

After working a bit with network namespaces—and knowing that OpenStack Neutron uses network namespaces in certain configurations, especially to support overlapping IP address spaces—I wondered how one might go about integrating multiple network namespaces into a broader configuration using OVS and GRE tunnels. Could I use VLANs to multiplex traffic from multiple namespaces across a single GRE tunnel?

To test my ideas, I came up with the following design:

As you can see in the diagram, my test environment has two KVM hosts. Each KVM host has a network namespace and a running guest domain. Both the network namespace and the guest domain are connected to an OVS bridge; the network namespace via a veth pair and the guest domain via a vnet port. A GRE tunnel between the OVS bridges connects the two hosts.

The idea behind the test environment was that the VM on one host would communicate with the veth interface in the network namespace on the other host, using VLAN-tagged traffic over a GRE tunnel between them.

Let’s walk through how I built this environment to do the testing.

I built KVM Host 1 using Ubuntu 12.04.2, and installed KVM, libvirt, and OVS. On KVM Host 1, I built a guest domain, attached it to OVS via a libvirt network, and configured the VLAN tag for its OVS port with this command:

ovs-vsctl set port vnet0 tag=10

In the guest domain, I configured the OS (also Ubuntu 12.04.2) to use the IP address

Also on KVM Host 1, I created the network namespace, created the veth pair, moved one of the veth interfaces, and attached the other to the OVS bridge. This set of commands is what I used:

ip netns add red
ip link add veth0 type veth peer name veth1
ip link set veth1 netns red
ip netns exec red ip addr add dev veth1
ip netns exec red ip link set veth1 up
ovs-vsctl add-port br-int veth0
ovs-vsctl set port veth0 tag=20

Most of the commands listed above are taken straight from the network namespaces article I wrote, but let’s break it down anyway just for the sake of full understanding:

  • The first command adds the “red” namespace.
  • The second command creates the veth pair, creatively named veth0 and veth1.
  • The third command moves veth1 into the red namespace.
  • The next two commands add an IP address to veth1 and set the interface to up.
  • The last two commands add the veth0 interface to an OVS bridge named br-int, and then set the VLAN tag for that port to 20.

When I’m done, I’m left with KVM Host 1 running a guest domain on VLAN 10 and a network namespace on VLAN 20. (Do you see how I got there?)

I repeated the process on KVM Host 2, installing Ubuntu 12.04.2 with KVM, libvirt, and OVS. Again, I built a guest domain (also running Ubuntu 12.04.2), configured the operating system to use the IP address, attached it to OVS via a libvirt network, and configured its OVS port:

ovs-vsctl set port vnet0 tag=20

Similarly, I also created a new network namespace and pair of veth interfaces, but I configured them as a “mirror image” of KVM Host 1, reversing the VLAN assignments for the guest domain (as shown above) and the network namespace:

ip netns add blue
ip link add veth0 type veth peer name veth1
ip link set veth1 netns blue
ip netns exec blue ip addr add dev veth1
ip netns exec blue ip link set veth1 up
ovs-vsctl add-port br-int veth0
ovs-vsctl set port veth0 tag=10

That leaves me with KVM Host 2 running a guest domain on VLAN 20 and a network namespace on VLAN 10.

The final step was to create the GRE tunnel between the OVS bridges. However, after I established the GRE tunnel, I configured the GRE port to be a VLAN trunk using this command (this command was necessary on both KVM hosts):

ovs-vsctl set port gre0 trunks=10,20,30

So I now had the environment I’d envisioned for my testing. VLAN 10 had a guest domain on one host and a veth interface on the other; VLAN 20 had a veth interface on one host and a guest domain on the other. Between the two hosts was a GRE tunnel configured to act as a VLAN trunk.

Now came the critical test—would the guest domain be able to ping the veth interface? This screen shot shows the results of my testing; this is the guest domain on KVM Host 1 communicating with the veth1 interface in the separate network namespace on KVM Host 2:

Success! Although not shown here, I also tested all other combinations as well, and they worked. (Note you’d have to use ip netns exec ping … to ping from the veth1 interface in the network namespace.) I now had a configuration where I could integrate multiple network namespaces with GRE tunnels and OVS. Unfortunately—and this is where the whole “technically interesting but practically useless” statement comes from—this isn’t really a usable configuration:

  • The VLAN configurations were manually applied to the OVS ports; this means they disappeared if the guest domains were power-cycled. (This could be fixed using libvirt portgroups, but I hadn’t bothered with building them in this environment.)
  • The GRE tunnel had to be manually established and configured.
  • Because this solution uses VLAN tags inside the GRE tunnel, you’re still limited to about 4,096 separate networks/network namespaces you could support.
  • The entire process was manual. If I needed to add another VLAN, I’d have to manually create the network namespace and veth pair, manually move one of the veth interfaces into the namespace, manually add the other veth interface to the OVS bridge, and manually update the GRE tunnel to trunk that VLAN. Not very scalable, IMHO.

However, the experiment was not a total loss. In figuring out how to tie together network namespaces and tunnels, I’ve gotten a better understanding of how all the pieces work. In addition, I have a lead on an even better way of accomplishing the same task: using OpenFlow rules and tunnel keys. This is the next area of exploration, and I’ll be sure to post something when I have more information to share.

In the meantime, feel free to share your thoughts and feedback on this post. What do you think—technically interesting or not? Useful in a real-world scenario or not? All courteous comments (with vendor disclosure, where applicable) are welcome.

Tags: , , , , , , ,

« Older entries