Linux Containers via LXC and Libvirt

One of the cool things about libvirt is the ability to work with multiple hypervisors and virtualization technologies, including Linux containers using LXC. In this post, I’m going to show you how to use libvirt with LXC, including leveraging libvirt to help automate attaching containers to Open vSwitch (OVS).

If you aren’t familiar with Linux containers and LXC, I invite you to have a look at my introductory post on Linux containers and LXC. It should give you enough background to make this post make sense.

To use libvirt with an LXC container, there are a couple of basic steps:

  1. Create the container using standard LXC user-space tools.
  2. Create a libvirt XML definition for the container.
  3. Define the libvirt container domain.
  4. Start the libvirt container domain.

The first part, creating the container, is pretty straightforward:

lxc-create -t ubuntu -n cn-02

This creates a container using the Ubuntu template and calls it cn–01. As you may recall from my introductory LXC post, this creates the container’s configuration and root filesystem in /var/lib/lxc by default. (I’m assuming you are using Ubuntu 12.04 LTS, as I am.)

Once you have the container created, you next need to get it into libvirt. Libvirt uses a standard XML-based format for defining VMs, containers, networks, etc. At first, I thought this might be the most difficult section, but thanks to this page I was able to create a template XML configuration.

Here’s the template I was able to create:

(If you can’t see the embedded code above, please click here.)

Simply take this XML template and save it as something like lxc-template.xml or similar. Then, after you’ve created your container using lxc-create as above, you can easily take this template and turn it into a specific container configuration with only one command. For example, suppose you created a container named “cn–02″ (as I did with the command I showed earlier). If you wanted to customize the XML template, just use this simple Unix/Linux command:

sed 's/REPLACE/cn-02/g' lxc-template.xml > cn-02.xml

Once you have a container-specific libvirt XML configuration, then defining it in libvirt is super-easy:

virsh -c lxc:// define cn-02.xml

Then start the container:

virsh -c lxc:// start cn-02

And connect to the container’s console:

virsh -c lxc:// console cn-02

When you’re done with the container’s console, press Ctrl-] (that’s Control and right bracket at the same time); that will return you to your host.

Pretty handy, eh? Further, since you’re now controlling your containers via libvirt, you can leverage libvirt’s networking functionality as well—which means that you can easily create libvirt virtual networks backed by OVS and automatically attach containers to OVS for advanced networking configurations. You only need to create an OVS-backed virtual network like I describe in this post on VLANs with OVS and libvirt.

I still need to do some additional investigation and testing to see how the networking configuration in the container’s config file interacts with the networking configuration in the libvirt XML file. For example, how do you define multiple network interfaces? Can you control the name of the veth pairs that show up in the host? I don’t have any answers for these questions (yet). If you know the answers, feel free to speak up in the comments!

All courteous feedback and interaction is welcome, so I invite you to start (or join) the discussion via the comments below.

Tags: , , , , , ,

  1. Alessandro’s avatar

    Very interesting! You mean by:

    1. Create the container using standard LXC user-space tools.

    including docker? I’m going to explore the use of virt-manager to manage running docker containers, this looks cool :)

  2. Sergio’s avatar

    Just one note, if you try to start the container and have any issue regarding the network:

    # virsh -c lxc:// start cn-02
    error: Failed to start domain cn-02
    error: internal error: Network ‘default’ is not active.

    Enable the network first
    # virsh
    virsh # net-start default
    Network default started

    # virsh -c lxc:// start cn-02
    Domain cn-02 started

    Nice write up !!

  3. slowe’s avatar

    Alessandro, I was more referring to tools like lxc-create, lxc-start, and the other lxc-* tools.

  4. Jack’s avatar

    I wonder know why I failed to start the LXC I defined with the XML you posted with the error cann’t find the source dir /var/lib/lxc/REPLACE/rootfs.
    Hope for your reply,thank you!

  5. slowe’s avatar

    Jack, you need to replace the word “REPLACE” with the appropriate value for the particular container you’re trying to start.

  6. Priya’s avatar

    Hi Scott,

    I tried to do this exactly the way you described, but after I created the container with lxc-create and define the container xml file, it gives me this error:

    lxc-create -t ubuntu -n cn-name

    root@/scripts# virsh -c lxc:// define cn.xml
    error: Failed to define domain from cn.xml
    error: operation failed: domain ‘cn-name’ already exists with uuid 42e4d176-eddb-42a2-95f2-2d6448b787c3

    Any idea why?

  7. slowe’s avatar

    Priya, don’t use lxc-create but use libvirt instead with the XML configuration.

  8. Priya’s avatar

    Scott, thanks, but in your post above, I thought you have said first create the container with lxc-create? I did use libvirt with the XML configuration, here is s snippet of my cn.xml file:




  9. Priya’s avatar

    Btw, your site is awesome, great job!!

  10. Gaurav bhambri’s avatar

    Good start to understand the basics of Container technology.


Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>