Working with KVM Guests

As you may already know, my new role at EMC has me spending more time on open source projects like KVM, OpenStack, and others. Over the past few days, I’ve been working with KVM for a bit, and I wanted to share a few things I’ve learned along the way. Readers who are very familiar with KVM won’t likely find anything new here (although I do encourage you to share any additional information or insight in the comments).

The information presented here is based on working with KVM using libvirt. For those unfamiliar with libvirt, it is an open source framework/API that can help manage multiple hypervisors and virtual machines. If you use a different management layer—such as oVirt—then things might look a bit different, so keep that in mind. (My use of libvirt vs. oVirt is not an endorsement one way or another; I just happened to start with libvirt.) Also, I’m only focusing on command-line tools here; there are a number of graphical user interfaces (GUIs) available as well, which I may investigate further at some point in the future.

The Components of a KVM Guest

Let’s start with looking at the different components of a KVM guest. If you are familiar with VMware virtualization, you know that a VMware-based VM has essentially two components:

  • The VM definition, stored in a .VMX file
  • The VM’s storage, typically stored in one or more .VMDK files

From what I’ve been able to determine so far, KVM guests also have essentially two components:

  • The VM definition, defined in XML format
  • The VM’s storage, stored either as a volume managed by an LVM or as a file stored in a file system

You can look at the XML configuration of a KVM guest (more properly referred to as a domain) in a couple of different ways. Both involve the use of virsh, the command-line tool that is part of libvirt:

  • To edit the configuration of a guest, use virsh edit <Name of guest VM> and the system will open the XML configuration in your default editor.
  • To export the configuration of a guest, use virsh dumpxml <Name of guest VM>. This dumps the XML configuration to STDOUT; you can redirect the output to a file if you like.

Here’s a snippet of the XML configuration for a KVM guest:

...
    <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/vm01.img'/>
      <target dev='hda' bus='ide'/>
      <address type='drive' controller='0' bus='0' unit='0'/>
    </disk>
...

The second component of a KVM guest is the storage; as I mentioned earlier, this can be a file on a file system or it can be a volume managed by a logical volume manager (LVM). The XML snippet above shows the configuration for a disk image stored as a file on a file system. This particular disk image is in QEMU raw format.

With that (very brief) overview complete, here is how you perform some common tasks with KVM guests.

Creating a KVM Guest

There are a couple of different ways to create a KVM guest:

  • Manually create the XML definition of the guest, then use virsh define <Name of XML file> to import the definition. You could, naturally, create a new XML definition based on an existing definition and just change a few parameters.
  • Use a libvirt-compatible tool, like virt-install, to create the guest definition.

Here’s a quick example of creating a KVM guest using virt-install (I’ve inserted backslashes and line breaks for readability):

virt-install --name vmname --ram 1024 --vcpus=1 \
--disk path=/var/lib/libvirt/images/vmname.img,size=20 \
--network bridge=br0 \
--cdrom /var/lib/libvirt/images/os-install.iso \
--graphics vnc --noautoconsole --hvm \
--os-variant win2k3

This command creates a KVM guest named “vmname”, with 1024 MB of RAM, a single vCPU, a 20 GB virtual disk (in the default QEMU raw format, thin provisioned so that not all 20 GB are allocated up front), connected to an OS installation ISO at the path specified and using hardware virtualization. The network connection is to a bridge called br0, which might be a fake bridge on Open vSwitch (which in my case it is). Access to the console is handled via VNC.

Full details on the various parameters for virt-install are available via the man page (or you can look here).

Starting and Stopping KVM Guests

This one is easy:

  • To start a VM, just run virsh start <Name of guest VM> and you’re off to the races.
  • To stop a VM, run virsh shutdown <Name of guest VM> and the guest will start an orderly shutdown. (BTW, I know that the orderly shutdown works for Ubuntu and Windows guests, but I haven’t figured out the exact mechanism yet. Any insight there?)

Changing Guest Configuration

If the VM is shut down, you can use the virsh edit <Name of guest VM> command to edit the XML definition directly.

If the VM is running, then only certain things can be done. You are supposed to be able to swap floppy or CD/DVD images using the virsh qemu-monitor-command command. In practice I found that it ejected CD/DVD images fine, but wouldn’t load a new CD/DVD ISO image. I’m still working on a fix for that one (tips are welcome).

Using Paravirtualized Drivers

For improved performance, you can also use paravirtualized drivers. (This is the equivalent of using VMXNET3 or PVSCSI in a VMware world.) So far I’ve only tested the network drivers on Ubuntu and Windows, but they seem to work just fine. (These paravirtualized drivers are referred to as the virtio drivers.)

To use virtio drivers for networking, edit the guest configuration like this:

<interface type='bridge'>
  <mac address='52:54:00:a0:55:ef'/>
  <source bridge='br0'/>
  <model type='virtio'/> (add this line)
</interface>

I found this page to be quite helpful in determining exactly how to enable the virtio network drivers. This part is the same for both Ubuntu and Windows guests; for Windows guests, you also have to install the virtio drivers into Windows. That can be a bit more problematic; I’d recommend copying them across the network before making the change above.

This change must be made while the guest is shut down, by the way.

Cold Migrations of KVM Guests

You can probably figure out how this one works by now:

  1. Shut down the guest using virsh shutdown <Name of guest VM>.
  2. Export the XML configuration using virsh dumpxml <Name of guest VM> > <Name of XML file>.
  3. Copy the XML definition you just created and the guest’s disk image (specified in the XML configuration) to the target node.
  4. Edit the XML configuration (as needed) to reflect any changes in paths.
  5. Define the guest on the new node using virsh define <Name of XML file>.

Obviously, this assumes an image-based disk, not a LVM-backed disk.

While hardly a comprehensive list of all the various operations that might need to be done with a KVM guest, hopefully this will at least get you started. I’ll post more as I learn more, and readers are encouraged to share tips, tricks, or other information in the comments below.

Tags: , , , ,

  1. Chris Cowley’s avatar

    Minor correction, but there is no way are you endorsing libvirt over oVirt. oVirt sits on top of libvirt and is comparable to vSphere. Libvirt is more comparable to the vKernel I would say.

    In terms of backend storage, there is far more to it than those 2 components aswell. The virtual disk could be (for example) a LUN on a SAN target. It could be a physical hard disk. It could even be a Ceph RADOS Block Device (not tried that myself yet).

    Only read the first few paragraphs so far, I will continue my constructive critism later :)

  2. Chris Cowley’s avatar

    I would also recommend you look at virt-manager now that you have got through the basics (sudo apt-get install virt-manager). Add your own use to the kvm group to save on pop up windows ;)

  3. slowe’s avatar

    Chris, thanks for your comments so far. As I stated in the post, my discussion on using libvirt was *NOT* an endorsement one way or another; it’s simply where I started. I’m sure that some posts about libvirt and oVirt will find their way here at some point. As for virt-manager, I’m still trying to figure out how to make that work. There is no GUI on my KVM host, so…not really sure how that would help. Feel free to enlighten me.

  4. Chris Cowley’s avatar

    On Ubuntu, as long as virt-install is working, it should just work. It may well be the serial output from the guest is not going to the right place. Are you able to create a new guest within virt-manager?

    Feel free to email me offline with any output/errors and I’ll have a look. Unfortunately I am in the middle of an international house move, so I cannot prmoise to get back to you in a timely fashion though

  5. Kyriakos Oikonomakos’s avatar

    (Re: migrations) You can always take a look at the virsh ‘managedsave’ option which allows you to suspend your system, dump it into an image and restore from where you were. I never tested it on different physical hosts but I don’t see why it shouldn’t work.

    PS: this is also promising: http://docs.fedoraproject.org/en-US/Fedora/13/html/Virtualization_Guide/sect-Virtualization-KVM_live_migration-Live_KVM_migration_with_virsh.html

  6. Joe Sanchez’s avatar

    Hi Scott,

    It’s been a while since my last comment but I thought this post was confirmation of a post I wrote over the weekend.

    http://www.vminstall.com/life-beyond-vsphere-openstack-smartcloud-cloudstack/

    I think VMware enthusiasts should start broadening their virtualization skills beyond VMware and ESX, and do it fast! Glad to so your post on KVM…

    A lot has changed since we started our blogs, and VMware was really the best of brand for virtualization.

    Do you plan on writing any KVM books?

    Thanks,
    Joe

  7. Chris Bennett’s avatar

    Hi Scott,

    “As for virt-manager, I’m still trying to figure out how to make that work. There is no GUI on my KVM host, so…not really sure how that would help.”

    The nice thing about Linux & X11 is you don’t need a GUI on your KVM host to use virt-manager. You can do two things:

    - install an X server on your desktop PC / management host. Use ‘X11 Forwarding’ over SSH. Your $DISPLAY variable should be set appropriately so when you run ‘virt-manager’, the content is sent to your X11 server.

    - alternatively, install virt-manager on another Linux host (say a ‘management’ Ubuntu VM). virt-manager supports communicating to libvirtd over SSH. You can actually attach multiple KVM hosts via this method and manage things similar to how vCenter managed multiple ESX hosts or a cluster…

    There are some benefits to running virt-manager locally on the KVM host, since you’ll be able to browse to local file resources without needing the resources in libvirt storage pools. This is where things are a little unpolished in the virt-manager space, but it’s always improving… :)

    Hope that helps.

    Chris

  8. Anton Coleman’s avatar

    Scott,

    If your still on a hang up on how to get virt-manager working without a GUI, look into running X11 on your host and setup X11 forwarding through a terminal client like putty or Mac OS X terminal in conjunction with either Xming on Windows or Xquartz on Mac (note if on a Mac you’ll want to enable the native X11 forwarding by editing your sshd_config file). The other alternative to using virt-manager is setting up a linux desktop client like ubuntu, fedora, mint, etc… Then install dirt-manager on that desktop and establish a remote connection to the kvm server you want to manage. I’ll create another comment speaking more in depth about my experiences with kvm, juju, openstack, and other kvm management tools.

  9. Anton Coleman’s avatar

    My apologies…I did not notice the post by Chris Bennett. Maybe I thought sounding redundant would help it sink in…

  10. Anton Coleman’s avatar

    Here is amazing site with a list of pretty much any management tool related to KVM technology.

    http://www.linux-kvm.org/page/Management_Tools

    The particular management tool I chose to evaluate first to help manage my test KVM environment was Convirt. You can do an apt-get for convirt on debian/ubuntu, but my suggestion would be to install it through the use of the tar ball. Convirt is closely related to oVirt, but there are some differences like the use of the spice protocol on oVirt. Convirt has an enterprise level version which includes high availability, better hardware monitoring, and obviously support.

  11. Chris Bennett’s avatar

    No problems Anton – you supplied some more info on Xquartz which I know nothing about so between the two of us, Scott will get the idea :)

  12. George’s avatar

    “virsh shutdown” sends an ACPI shutdown call to the virtual machine.
    If the VM has ACPI support then it will nicely shutdown.

  13. Sriram S’s avatar

    Hi Scott

    In the call to virt-install you have used br0 (fake bridge). I have two question related to networking with KVM. 1. How is IP address allocated to the guest VM? 2. If the bridge br0 was on my wireless LAN, will the guest have access to the wireless network?

    Sriram

  14. Amine’s avatar

    Thanks Scott for your post.

    I had worked on setting up KVM on Ubuntu and running Linux and Windows guests. I am wondering if there is an api which could be used to run programs in the guest vms or copying files from host to guest like what the vix api does in vmare esxi.

    Thanks,

  15. John’s avatar

    I am also interested in running and copying from/to guest VMs like VIX API. Anyone know if there is a solution or is planned?

  16. slowe’s avatar

    John, I think that the libguestfs project might offer the kind of functionality you’re seeking, but I don’t know that for certain.

Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>