Virtualization

You are currently browsing articles tagged Virtualization.

Using VLANs with OVS and libvirt

Wednesday, November 7, 2012 in Interoperability, Networking, Virtualization by slowe | 2 comments

In previous posts, I’ve shown you how to use Open vSwitch (OVS) with VLANs through fake bridges, as well as how to wrap libvirt virtual network around OVS fake bridges. Both of these techniques are acceptable for configuring VLANs with OVS, but in this post I want to talk about using VLANs with OVS via a greater level of libvirt integration. This has been talked about elsewhere, but I wasn’t able to make it work until libvirt 1.0.0 was released. (Update: I was able to make it work with an earlier version. See here.)

First, let’s recap what we know so far. If you know the port to which a particular domain (guest VM) is connected, you can configure that particular port as a VLAN trunk like this:

ovs-vsctl set port <port name> trunks=10,11,12

This configuration would pass the VLAN tags for VLANs 10, 11, and 12 all the way up to the domain, where—assuming the OS installed in the domain has VLAN support—you could configure network connectivity appropriately. (I hope to have a blog post up on this soon.)

Along the same lines, if you know the port to which a particular domain is connected, you could configure that port as a VLAN access port with a command like this:

ovs-vsctl set port <port name> tag=15

This command makes the domain a member of VLAN 15, much like the use of the switchport access vlan 15 command on a Cisco switch. (I probably don’t need to state that this isn’t the only way—see the other OVS/VLAN related posts above for more techniques to put a domain into a particular VLAN.)

These commands work perfectly fine and are all well and good, but there’s a problem here—the VLAN information isn’t contained in the domain configuration. Instead, it’s in OVS, attached to an ephemeral port—meaning that when the domain is shut down, the port and the associated configuration disappears. What I’m going to show you in this post is how to use VLANs with OVS in conjunction with libvirt for persistent VLAN configurations.

This document was written using Ubuntu 12.04.1 LTS and Open vSwitch 1.4.0 (installed straight from the Precise Pangolin repositories using apt-get). Libvirt was compiled manually (see instructions here). Due to some bugs, it appears you need at least version 1.0.0 of libvirt. Although the Silicon Loons article I referenced earlier mentions an earlier version of libvirt, I was not able to make it work until the 1.0.0 release. Your mileage may vary, of course—I freely admit that I might have been doing something wrong in my earlier testing.

To make VLANs work with OVS and libvirt, two things are necessary:

  1. First, you must define a libvirt virtual network that contains the necessary portgroup definitions.
  2. Second, you must include the portgroup reference to the virtual network in the domain (guest VM) configuration.

Let’s look at each of these steps.

Creating the Virtual Network

The easiest way I’ve found to create the virtual network is to craft the network XML definition manually, then import it into libvirt using virsh net-define.

Here’s some sample XML code (I’ll break down the relevant parts after the code):

The key takeaways from this snippet of XML are:

  1. First, note that the OVS bridge is specified as the target bridge in the <bridge name=...> element. You’ll need to edit this as necessary to make your specific OVS configuration. For example, in my configuration, ovsbr0 refers to a bridge that handles only host management traffic.
  2. Second, note the <portgroup name=...> element. This is where the “magic” happens. Note that you can have no VLAN element (as in the vlan-01 portgroup), a VLAN tag (as in the vlan-10 or vlan-20 portgroups), or a set of VLAN tags to pass as a trunk (as in the vlan-all portgroup).

Once you’ve got the network definition in the libvirt XML format, you can import that configuration with virsh net-define <XML filename>. (Prepend this command with sudo if necessary.)

After it is imported, use virsh net-start <network name> to start the libvirt virtual network. If you make changes to the virtual network, such as adding or removing portgroups, be sure to restart the virtual network using virsh net-destroy <network name> followed by virsh net-start <network name>.

Now that the virtual network is defined, we can move on to creating the domain configuration.

Configuring the Domain Networking

As far as I’m aware, to include the appropriate network definitions in the domain XML configuration, you’ll have to edit the domain XML manually.

Here’s the relevant snippet of domain XML configuration:

You’ll likely have more configuration entries in your domain configuration, but the important one is the <source network=...> element, where you’ll specify both the name of the network you created as well as the name of the portgroup to which this domain should be attached.

With this configuration in place, when you start the domain, it will pass the necessary parameters to OVS to apply the desired VLAN configuration automatically. In other words, once you define the desired configuration in the domain XML, it’s maintained persistently inside the domain XML (instead of on the ephemeral port in OVS), re-applied anytime the domain is started.

Verifying the Configuration

Once the appropriate configuration is in place, you can see the OVS configuration created by libvirt when a domain is started by simply using ovs-vsctl show or—for more detailed information—ovs-vsctl list port <port name>. Of particular interest when using ovs-vsctl list port <port name> are the tag and/or trunks values; these are where VLAN configurations are applied.

Summary

In this post, I’ve shown you how to create libvirt virtual networks that integrate with OVS to provide persistent VLAN configurations for domains connected to an OVS bridge. The key benefit that arises from this configuration is that you longer need to know to which OVS port a given domain is connected. Because the VLAN configuration is stored with the domain and applied to OVS automatically when the domain is started, you can be assured that a domain will always be attached to the correct VLAN when it starts.

As usual, I encourage your feedback on this article. If you have questions, thoughts, corrections, or clarifications, you are invited to speak up in the comments below.

Tags: , , , , , ,

A Quick Warning About Compiling libvirt

Tuesday, November 6, 2012 in Interoperability, Linux, Virtualization by slowe | 2 comments

If you’ve been reading my site for any length of time, you know that I’ve been working with libvirt, the open source virtualization API, for a while. As part of my testing with libvirt, I’ve needed to stay on the “cutting edge” of libvirt’s development (so to speak), so I’ve been compiling my own version of libvirt from the source packages available from the libvirt HTTP server.

You can see some of the posts I’ve written about compiling libvirt:

Compiling libvirt 1.0.0 on Ubuntu 12.04 and 12.10
Compiling libvirt 0.10.1 on Ubuntu 12.04
Compiling libvirt 0.10.1 on CentOS 6.3

While these instructions work, there is an important consideration to keep in mind here: updating the system. When you compile and install your own binaries, the system has no way of knowing how to apply updates. For example, on Ubuntu, this means that apt-get, aptitude, and Update Manager are not aware that a newer version of libvirt is present on the system. Thus, when you go to update the system—you do keep your system updated, right?—then things will break. (Note that there are workarounds for apt-get and aptitude, but it does not appear that these workarounds also work with Update Manager.)

If you are considering following my instructions to compile your own binaries for libvirt, please be sure to keep this consideration in mind. This sort of “gotcha” might be fine for a small home-based lab like mine, but it probably isn’t the sort of issue you’d like to introduce into any sort of production (or quasi-production) environment.

Thanks to Theron Conrey for highlighting the significance of this concern. Be sure to check out Theron’s comment on this article for more information on a potential Ubuntu PPA that might help with this issue.

Tags: , , , ,

Compiling libvirt 1.0.0 on Ubuntu 12.04 and 12.10

Monday, November 5, 2012 in Interoperability, Linux, Virtualization by slowe | 15 comments

In case you hadn’t noticed late last week, the open source virtualization API project libvirt released version 1.0.0—a major milestone, obviously. To celebrate the 1.0.0 release, here are instructions for compiling the libvirt 1.0.0 release on Ubuntu 12.04 and 12.10.

(If you need instructions on compiling an earlier release of libvirt on Ubuntu, see this post. It works for libvirt 0.10.1 and 0.10.2 on Ubuntu 12.04 LTS; I haven’t yet tested it on Ubuntu 12.10.)

These instructions assume that you have built a plain-jane Ubuntu system (either 12.04 LTS or 12.10). I tested these instructions on an Ubuntu 12.04.1 LTS VM, freshly built and updated using apt-get update && apt-get upgrade, on a freshly-built Ubuntu 12.01 VM (also up to date), and a physical system running an up-to-date instance of Ubuntu 12.04.1 that also had KVM and an earlier version of libvirt installed. These instructions worked in all three cases.

Ready? Let’s get started!

  1. Download the libvirt-1.0.0.tar.gz tarball from the libvirt.org HTTP server.

  2. Extract the tarball using tar -xvzf libvirt-1.0.0.tar.gz.

  3. Using apt-get, ensure that the following packages are installed: gcc, make, pkg-config (already installed in my testing), libxml2-dev, libgnutls-dev, libdevmapper-dev, libcurl4-gnutls-dev, and python-dev. (Interestingly enough, earlier versions also required libyajl-dev, but this version didn’t seem to need it.) You can probably omit the libcurl4-gnutls-dev package if you don’t want ESX support; I did want ESX support, so I included it.

  4. From within the extracted libvirt-1.0.0 directory, run the configure command. I specified particular directories because that’s where the prebuilt binaries from Ubuntu are normally placed. I’ve line-wrapped it here for readability:

    5. ./configure --prefix=/usr --localstatedir=/var \
--sysconfdir=/etc --with-esx=yes

  5. The above command does not include Xen support; if you need Xen support, you’ll need to add --with-xen=yes to the command, and your list of prerequisite packages will change (there are additional packages you’ll need to install).

  6. Once the configure command completes, then complete the process with make and make install. Note that I used sudo make install here because I’m installing files into privileged system locations.

    7. make
sudo make install

  7. Finally, to verify that everything seems to be working as expected, restart libvirt with initctl stop libvirt-bin and initctl start libvirt-bin. (Yes, you could use initctl restart, but this allows you to see the clean stop and start of the libvirt daemon.) As an aside, note that this step only works if you either a) had a previous version of libvirt installed, or b) create the initctl job for libvirt. I chose the first approach.

  8. As a final verification step, run virsh --version or libvirtd --version to verify that you’re running libvirt 1.0.0.

That’s it! Now, as others have pointed out, this will create some potential system administration challenges, in that apt-get will still suggest new libvirt packages to install when you try to update the system. I think that there are some commands you can run to keep the manually compiled version instead of the version apt-get is suggesting, but I haven’t yet determined exactly which commands to use. (If you have more information, please speak up in the comments!)

I have a series of new libvirt-related blog posts in the works, so stay tuned for those. In the meantime, feel free to post any questions, corrections, or clarifications in the comments below. Courteous comments are always welcome.

Tags: , , , ,

Technology Short Take #26

Monday, October 29, 2012 in Linux, Macintosh, Networking, Storage, UNIX, Virtualization by slowe | No comments

Welcome to Technology Short Take #26! As you might already know, the Technology Short Takes are my irregularly-published collections of links, articles, thoughts, and (sometimes) rants. I hope you find something useful here!

Networking

  • Chris Colotti, as part of a changed focus in his role at VMware, has been working extensively with Nicira NVP. He’s had a couple of good posts; this one is a primer on how NVP works, and this one discusses the use of the Open vSwitch (OVS) vApp. As I mentioned before in other posts, OVS is popping up in more and more places—it might be a good idea to make sure you’re familiar with it.
  • This article by Ivan Pepelnjak on VXLAN termination on physical devices is over a year old, but still very applicable—especially considering Arista Networks recently announced their 7150S switch, which sports hardware VTEP (VXLAN Tunnel End Point) support (meaning that it can terminate VXLAN segments).
  • Brad Hedlund dives into Midokura Midonet in this post on L2-L4 network virtualization. It’s a good overview (thanks Brad!) and worth reading if you want to get up to speed on what Midokura is doing. (Oh, just as an aside: note that Midokura leverages OVS in their solution. Just saying…)
  • This blog post provides more useful information from Kamau Wanguhu on VXLAN and proxy ARP. Kamau also has an interesting post on network virtualization, although—to be honest—the post is long on messaging/positioning and short on technical information. I prefer the latter instead of the former.

Servers/Hardware

  • This mention of the Dell PowerEdge M I/O Aggregator looks interesting, although I’m still not real clear on exactly what it is or how it works. I guess this first article was a tease?

Security

Nothing this time around, but I’ll stay alert for items to include in future posts!

Cloud Computing/Cloud Management

  • Want to know a bit more about how to configure VXLAN inside VCD? Rawlinson Rivera has a nice write-up that is worth reviewing.
  • Clint Kitson, an EMC vSpecialist, talks about some VCD integrity scripts he created. Looks like some pretty cool stuff—great work, Clint!
  • For the past couple of weeks I’ve been (slowly) reading Kevin Jackson’s OpenStack Cloud Computing Cookbook; it’s very useful. It’s worth a read if you want to get up to speed on OpenStack; naturally, you can get it from Amazon.

Operating Systems/Applications

  • At the intersection of cloud-based storage and configuration management, I happened to find this very interesting Puppet module designed to fetch and update files from an S3 bucket. Through this module, you could store files in S3 instead of using Puppet’s built-in file server. (By the way, this module also works with OpenStack Swift as well.)
  • One of the things I’ve complained about regarding newer versions of OS X is the “hiding” of the Unix underpinnings. Perhaps I should read this book and see if my thinking is unfounded?

Storage

  • Chris Evans takes a look at Hyper-V 3.0′s Virtual Fibre Channel feature in this write-up. From what I’ve read, it sounds like Hyper-V’s NPIV implementation is more robust than VMware’s broken and busted NPIV implementation. (If you don’t know why I say that about VMware’s implementation, ask anyone who’s tried to use it.) The real question is this: is NPIV support in a hypervisor of any value any longer?
  • Gina Minks (formerly of Dell, now with Inktank) recommended I have a look at Ceph and mentioned this post on migrating to Ceph (with a little libvirt thrown in).
  • Gluster might be another project that I need to spend some time examining; this post on using Gluster with oVirt 3.1 looks interesting. Anyone have any pointers for a Gluster beginner?
  • Mirantis has a post about some Nova Volume integration with Isilon. I’ve often said that I think scale-out platforms like Isilon (among others) are an important foundation for future storage solutions. It’s cool to see some third-party development happening to integrate Isilon and OpenStack.

Virtualization

That’s all for this time around. As always, courteous comments are welcome (encouraged, in fact!), so feel free to speak up in the comments below. I’d love to hear your feedback.

Tags: , , , , , , , , , , , , , ,

Link Aggregation and VLAN Trunking with Brocade FastIron Switches

Friday, October 26, 2012 in Interoperability, Networking, Virtualization by slowe | 4 comments

In this post, I’ll be sharing with you information on how to do link aggregation (with LACP) and VLAN trunking on a Brocade FastIron switch with both VMware vSphere as well as Open vSwitch (OVS).

Throughout the majority of my career, my networking experience has been centered around Cisco’s products. You can easily tell that from looking at the articles I’ve published here. However, I’ve recently had the opportunity to spend some time working with a Brocade FastIron switch (a 48-port FastIron Edge X, specifically, running software version 7.2), and I wanted to write-up what I’ve learned about how to do link aggregation and VLAN trunking in conjunction with both VMware vSphere as well as OVS.

Configuring Link Aggregation with LACP

When researching how to do link aggregation on a Brocade FastIron, I came across a number of different articles suggesting two different ways to configure link aggregation (ultimately I followed the information provided in this article and this article). I think that the difference in the configuration comes down to whether or not you want to use LACP, but I’m not completely sure. (If you’re a Brocade/Foundry expert, feel free to weigh in.)

To configure a link aggregate using LACP, use these commands:

  • You’ll use the link-aggregate configure key <unique key> command to identify which interfaces may participate in a given link aggregate. The key must range from 10000 to 65535, and has to be unique for each group of interfaces in a link aggregate bundle. The switch uses the key to identify which ports may be a part of a link aggregate.
  • You’ll use the link-aggregate active command to indicate the use of LACP for link aggregation configuration and negotiation.

For example, if you wanted to configure port 10 on a switch for link aggregation, the commands would look something like this:

switch(config)# interface ethernet 10
switch(config-if-e1000-10)# link-aggregate configure key 10000
switch(config-if-e1000-10)# link-aggregate active

For each additional port that should also belong in this same link aggregate bundle, you would repeat these commands and use the same key value. As I mentioned earlier, the identical key value is what tells the switch which interfaces belong in the same bundle.

Configuring the virtualization host is pretty straightforward from here:

  • If you are using vSphere, note that you’ll need to use vSphere 5.1 and a vSphere Distributed Switch (VDS) in order to use LACP. In order to use LACP, you’ll need to set your teaming policy to “Route based on IP hash,” and then you must enable LACP in the settings for the uplink group. Chris Wahl has a nice write-up here, including a list of the caveats of using LACP with vSphere. VMware also has a VMware KB article on the topic.
  • If you are using OVS, you can follow the instructions I provided in this post on link aggregation and LACP with Open vSwitch.

Configuring VLANs

Although VLANs are (generally) interoperable between different switch vendors due to the broad adoption of the 802.1Q standard, the details of each vendor’s implementation of VLANs is just different enough to make life difficult. In this particular case, since I learned Cisco’s VLAN implementation first, Brocade’s VLAN implementation on the FastIron Edge X series switches seemed rather odd. I’m sure that had I learned Brocade’s implementation first, then Cisco’s version would seem odd.

In any case, the commands you use for VLANs are as follows:

  • To create a VLAN, use the vlan <VLAN identifier> command.
  • To add a port to that VLAN, so that traffic across that port is tagged for the specified VLAN, use the tagged ethernet <interface> command.
  • To add a range of ports to a VLAN, use the tagged ethernet <start interface> to <end interface> command.
  • To allow a port to carry both untagged (native, or default VLAN) and tagged traffic, you must use the dual-mode command. Otherwise, a port carries only untagged or tagged traffic. (This was a key difference in Brocade’s VLAN implementation that threw me off at first.)

So, if you wanted to create VLAN 30, add Ethernet interface 24 to that VLAN, and configure the interface to carry both tagged and untagged traffic, the commands would look something like this:

switch(config)# vlan 30 name server-subnet
switch(config-vlan-30)# tagged ethernet 24
switch(config-vlan-30)# interface ethernet 24
switch(config-if-e1000-24)# dual-mode

Once the VLANs are created and the interfaces are added to the VLANs, configuring the virtualization hosts is—once again—pretty straightforward:

I hope this information is useful to someone. If anyone has any corrections or clarifications, I encourage you to add your information to the comments on this post.

Tags: , , , , , ,

Why Mess Around with libvirt?

Tuesday, October 23, 2012 in Interoperability, Virtualization by slowe | 3 comments

You might have noticed that I’ve been writing a fair number of articles around libvirt, the open source virtualization API project. Here are some of the libvirt-related articles that I’ve posted over the last couple of months (and there are more in the works):

Working with KVM Guests
Compiling libvirt 0.10.1 on CentOS 6.3
Compiling libvirt 0.10.1 on Ubuntu 12.04
Wrapping libvirt Virtual Networks Around Open vSwitch Fake Bridges

While libvirt is pretty cool in and of itself (I really like the abstraction layer that libvirt provides between multiple hypervisors and management tools farther up the stack), one of the primary reasons that I’m spending some time working with and attempting to understand libvirt is because libvirt is a key component in the communication that occurs between the OpenStack Compute (Nova) software and the underlying hypervisors that are supported with OpenStack (especially KVM). In attempting to educate myself about OpenStack, I’ve decided to first start by making myself very familiar with a few key components, including libvirt, Open vSwitch, and KVM. This provides a foundation upon which I can build additional knowledge.

At least, that’s the general idea. As always, I’m open to constructive feedback—is it worth the time, or not? I’d love to hear from others who might be on a similar path of discovery and education. Feel free to share your thoughts in the comments below.

Tags: , , ,

Wrapping libvirt Virtual Networks Around Open vSwitch Fake Bridges

Monday, October 22, 2012 in Interoperability, Networking, Virtualization by slowe | 1 comment

In a previous article, I talked about how to use Open vSwitch (OVS) fake bridges to bring VLAN support into your environment. In this article, I show you how to wrap a libvirt virtual network around your OVS fake bridge.

You might ask, “Why wrap a libvirt virtual network around an OVS fake bridge when you can just use the OVS bridge directly?” That’s a good question, and—to be perfectly honest—I don’t have a great answer. At first glance, it seems like it might make things easier if you are mixing both OVS-based networks and other types of networks, but I don’t know that for certain. If anyone has any feedback one way or the other (why this is a good idea or why it’s not a good idea), please speak up in the comments.

Now that we have that out of the way, the process for using a libvirt virtual network with an OVS fake bridge is actually pretty straightforward. First, create the appropriate OVS fake bridges using the instructions in this article. So, for example, you might create a fake bridge for VLAN 100 like this:

ovs-vsctl add-br vlan100 ovsbr0 100

Next, create an XML definition for a libvirt virtual network. For a fake bridge named vlan100, your XML definition might look something like this:

<network>
  <name>vlan100-net</name>
  <forward mode='bridge'/>
  <bridge name='vlan100'/>
</network>

Then, in the guest domain configuration, you reference the libvirt virtual network instead of the underlying bridge directly, like this:

<interface type='network'>
  <mac address='11:22:33:aa:bb:cc'/>
  <source network='vlan100-net'/>
  ...
</interface>

And that’s it! Based on my testing, it even appears that you can make this change on the fly, without having to reboot the guest domain. However, I could be wrong—if anyone knows definitively, please speak up in the comments. Any other corrections, clarifications, or questions are also welcome in the comments below.

Tags: , , , , ,

VLANs with Open vSwitch Fake Bridges

Friday, October 19, 2012 in Interoperability, Networking, Virtualization by slowe | 4 comments

In other posts, I’ve (briefly) talked about how to configure Open vSwitch (OVS) for use with VLANs. If you know the port to which a guest is connected, you can configure that particular port as a VLAN trunk like this:

ovs-vsctl set port <port name> trunks=10,11,12

This configuration would pass the VLAN tags for VLANs 10, 11, and 12 all the way up to the guest, where—assuming the OS installed in the guest has VLAN support—you could configure network connectivity appropriately.

Alternately, if you know the port to which a particular guest is connected, you could configure that port as a VLAN access port with a command like this:

ovs-vsctl set port <port name> tag=15

This command makes the guest a member of VLAN 15, much like the use of the switchport access vlan 15 command on a Cisco switch.

These commands are all well and good, but there’s a couple problems here:

  1. First, you must know which port corresponds to which guest domain. Thus far, I have been unable to determine what set of commands will help me (you) establish the mapping between ports/interfaces and guest domains. (If you know how, please speak up in the comments!)
  2. Second, even if you do know which port corresponds to which guest, the settings are ephemeral. That is, when you power off the guest, the port—and its associated configuration—goes away. You’d then need to reapply the configuration to the port when you start the guest domain again.

Clearly, this is not ideal. Fortunately, there is a workaround—a couple of them, actually. One workaround is to add OVS and VLAN support to libvirt (something that is actually mentioned here). This is a great idea—but it doesn’t work just yet. On some systems (I use Ubuntu 12.04.1 LTS with libvirt 0.10.2), the libvirt-OVS-VLAN integration causes an error. A patch has been submitted to libvirt to fix this problem (great work Kyle!), but it hasn’t (yet) made it into a release.

Without OVS/VLAN support in libvirt, we have only one other workaround: OVS fake bridges. OVS fake bridges look and act like a bridge, but are tied to a particular VLAN ID. (I haven’t seen/found a way to use a fake bridge to do VLAN trunking up to a guest domain. Anyone else know how?) In this post, I’m going to show you how to use OVS fake bridges to add VLAN support to your OVS environment.

This post was written using Ubuntu 12.04.1 LTS with Open vSwitch 1.4.0 (straight out of the Precise Pangolin repositories). Please note that the commands might be slightly different on other distributions or with other versions of OVS.

To create a fake bridge, you’ll use a modified form of the ovs-vsctl add-br command. The command is so subtly different that I missed it quite a few times when reading through the documentation for ovs-vsctl. Here’s the command you’ll need:

ovs-vsctl add-br <fake bridge> <parent bridge> <VLAN>

Let’s look at an example. Suppose you had an existing OVS bridge named ovsbr0, and you wanted to add a fake bridge to support VLAN 100. You would use this command:

ovs-vsctl add-br vlan100 ovsbr0 100

When you create (or edit) a guest domain, you’ll assign it to the new fake bridge (named vlan100 in this example). So, looking at the libvirt XML code for a guest domain, it might look something like this:

<interface type='bridge'>
  <mac address='11:22:33:aa:bb:cc'/>
  <source bridge='vlan100'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>

Naturally, you could also create a libvirt virtual network that corresponds to the fake bridge as well. (I’ll likely post a separate article around that idea.)

Then, when you powered up the guest domain and ran ovs-vsctl show, you’d see something like this:

Bridge "ovsbr0"
    Port "bond0"
        Interface "eth1"
        Interface "eth2"
    Port "ovsbr0"
        Interface "ovsbr0"
            type: internal
    Port "vnet0"
        tag: 100
        Interface "vnet0"
    Port "vlan100"
        tag: 100
        Interface "vlan100"
            type: internal

Note that the guest domain’s port/interface are automatically given the fake bridge’s VLAN tag, without any further interaction/configuration required by the user or administrator. Much better!

Assuming you’re using fake bridges (and if you’re using OVS and VLANs, I’m not sure how you wouldn’t be), there are a couple other commands you might find helpful as well:

  • The ovs-vsctl br-to-vlan command will print the VLAN ID for a given bridge. If the bridge is a real bridge, the command returns 0; if the bridge is a fake bridge, it returns the VLAN ID.
  • The ovs-vsctl br-to-parent command returns the parent bridge for a given fake bridge. If the specified bridge is a real bridge, it returns the real bridge.

Using fake bridges with link aggregation is also possible, as you can see from the snippet of OVS configuration above. More information on OVS with link aggregation is available here.

I hope this information is useful. OVS is a really powerful piece of software, and I’m enjoying learning more about it and how to use it. If anyone has any additional information, please feel free to speak up in the comments. All courteous comments are welcome!

Tags: , , , , ,

Link Aggregation and LACP with Open vSwitch

Friday, October 19, 2012 in Interoperability, Networking, Virtualization by slowe | 1 comment

In this post, I’m going to show you how to use link aggregation (via the Link Aggregation Control Protocol, or LACP) with Open vSwitch (OVS). First, though, let’s cover some basics.

In the virtualization space, it’s extremely common to want to use multiple physical network connections in your hypervisor hosts to support guest (virtual machine) traffic. The problem is that modern-day networking is—for now—largely constrained by the presence of Spanning Tree Protocol (STP), which limits the use of multiple connections between network devices (especially switches). Since most hypervisors have some form of virtual switch to support guest traffic, and since users don’t want to be constrained by STP the hypervisors have had to find workarounds.

VMware works around STP by causing their virtual switches to operate in what is called “end-host mode,” meaning that the virtual switch does not participate in STP (newer versions of vSphere can, in fact, block STP BPDUs from being emitted), the virtual switch does not forward frames received on one uplink back out another uplink, and traffic from VMs is statically assigned (pinned) to an uplink. (This behavior is, of course, configurable.) Because of these default behaviors, users in VMware environments simply connect multiple links to their hosts and off they go.

Other environments behave differently. Environments using Open vSwitch (OVS), for example, need to use other methods to work around the presence of STP, especially considering that OVS is more a full-featured virtual switch than the standard VMware vSwitch. In most cases, the workaround involves the use of link aggregation; specifically, the use of Link Aggregation Control Protocol (LACP), a standardized protocol that allows devices to automatically negotiate the configuration and use of link aggregates comprised of multiple physical links.

Now that you have the background, let’s dive into the details of how to make this work. These instructions on using LACP with OVS do make a few assumptions:

  1. First, I assume that OVS is already installed and working.
  2. I assume that the management traffic to/from your host is not running through OVS, and thus won’t be interrupted by any configurations you do here. If this is not the case, and you do have management traffic running through OVS, you might want to exercise some additional caution to ensure you don’t accidentally cut your connectivity to the host.
  3. I assume that you know how to configure your physical switch(es) to support LACP on the links coming in from OVS. The configuration will vary from switch vendor to switch vendor; refer to your vendor’s documentation for details.

This post was written using Ubuntu 12.04.1 LTS and Open vSwitch 1.4.0 (installed using apt-get directly from the Precise Pangolin repositories). The use of a different Linux distribution and/or a different version of OVS might make this process slightly different.

The first step is to add a bridge (substitute your desired bridge name for ovsbr1 in the following command):

ovs-vsctl add-br ovsbr1

Once the bridge is established, then you’ll need to create a bond. This is the actual link aggregate on OVS. The syntax for adding a bond looks something like this:

ovs-vsctl add-bond <bridge name> <bond name> <list of interfaces>

So, if you wanted to add a bond to ovsbr1 using physical interfaces eth1 and eth3, your command would look something like this:

ovs-vsctl add-bond ovsbr1 bond0 eth1 eth3

However, there’s a problem with this configuration: by default, LACP isn’t enabled on a bond. To fix this, you have two options.

  1. Change the command use to create the bond, so that LACP is enabled when the bond is created.
  2. Enable LACP after the bond is created.

For option #1, you’ll simply append lacp=active to the command to create the bond, like so:

ovs-vsctl add-bond ovsbr1 bond0 eth1 eth3 lacp=active

For option #2, you’d use ovs-vsctl set to modify the properties of the bond. Here’s an example:

ovs-vsctl set port bond0 lacp=active

Once the bond is created and LACP is enabled, you can check the configuration and/or status of the bond. Assuming that you’ve already configured your physical switch correctly, your bond should be working and passing traffic. You can use this command to see the status of the bond:

ovs-appctl bond/show <bond name>

The output from that command will look something like this:

bond_mode: balance-slb
bond-hash-algorithm: balance-slb
bond-hash-basis: 0
updelay: 0 ms
downdelay: 0 ms
next rebalance: 6415 ms
lacp_negotiated: true

slave eth4: enabled
    active slave
    may_enable: true

slave eth3: enabled
    may_enable: true

slave eth1: enabled
    may_enable: true

slave eth2: enabled
    may_enable: true

This command will show more detailed LACP-specific information:

ovs-appctl lacp/show <bond name>

This command returns a great deal of information; here’s a quick snippet:

---- bond0 ----
    status: active negotiated
    sys_id: 00:22:19:bd:db:dd
    sys_priority: 65534
    aggregation key: 4
    lacp_time: fast

slave: eth1: current attached
    port_id: 4
    port_priority: 65535

    actor sys_id: 00:22:19:bd:db:dd
    actor sys_priority: 65534
    actor port_id: 4
    actor port_priority: 65535
    actor key: 4
    actor state: activity timeout aggregation synchronized collectingdistributing

    partner sys_id: 00:12:f2:cc:6d:40
    partner sys_priority: 1
    partner port_id: 12
    partner port_priority: 1
    partner key: 10000
    partner state: activity aggregation synchronized collectingdistributing

You can also use this command to view the configuration details of the bond:

ovs-vsctl list port bond0

The output from this command will look something like this:

_uuid               : ae7eb7ca-e3e0-4166-bcfb-4348071799e0
bond_downdelay      : 0
bond_fake_iface     : false
bond_mode           : []
bond_updelay        : 0
external_ids        : {}
fake_bridge         : false
interfaces          : [9963381b-6a7d-4a8f-acf8-86150361530e,bee2df86-ed14-456b-8f3a-25fb00fa6040, daf5ac51-4135-4e3c-a937-c62dfc4b5e9f,fcd2d6ef-9a18-452a-9a79-1c97e5a95ef2]
lacp                : active
mac                 : []
name                : "bond0"
other_config        : {lacp-time=fast}
qos                 : []
statistics          : {}
status              : {}
tag                 : []
trunks              : []
vlan_mode           : []

In learning how to use LACP with OVS, I found this article to be extremely helpful.

If you have questions, or have additional information to share with me and/or other readers, please speak up in the comments. Thanks!

Tags: , , , ,

OpenStack Summit Day 1 Recap Links

Tuesday, October 16, 2012 in General, Networking, Storage, Virtualization by slowe | 1 comment

This is by no means a comprehensive list, but I did want to share a few resources recapping Day 1 of the OpenStack Summit that’s happening this week in San Diego. I’m positive there are additional resources that I’ve missed, so feel free to share more links and resources in the comments to this article.

Recap Posts

OpenStack Summit Day 1 Recap
OpenStack Summit – Day 1
OpenStack Summit: Day 1

News/Announcements Posts

OpenStack Design Summit and Conference Day 1 News Update
Announcing Chef for OpenStack Folsom
Cisco announces its own OpenStack distribution—what will the effect be on VMware?
Rackspace Launches Product Certification for Private Cloud Partners (note the quote from new EMC CTO John Roese)
Why Google Compute Engine for OpenStack

Additional resources and links are welcome in the comments. Thanks!

Tags: , ,

« Older entries § Newer entries »