Link Aggregation and LACP with Open vSwitch

In this post, I’m going to show you how to use link aggregation (via the Link Aggregation Control Protocol, or LACP) with Open vSwitch (OVS). First, though, let’s cover some basics.

In the virtualization space, it’s extremely common to want to use multiple physical network connections in your hypervisor hosts to support guest (virtual machine) traffic. The problem is that modern-day networking is—for now—largely constrained by the presence of Spanning Tree Protocol (STP), which limits the use of multiple connections between network devices (especially switches). Since most hypervisors have some form of virtual switch to support guest traffic, and since users don’t want to be constrained by STP the hypervisors have had to find workarounds.

VMware works around STP by causing their virtual switches to operate in what is called “end-host mode,” meaning that the virtual switch does not participate in STP (newer versions of vSphere can, in fact, block STP BPDUs from being emitted), the virtual switch does not forward frames received on one uplink back out another uplink, and traffic from VMs is statically assigned (pinned) to an uplink. (This behavior is, of course, configurable.) Because of these default behaviors, users in VMware environments simply connect multiple links to their hosts and off they go.

Other environments behave differently. Environments using Open vSwitch (OVS), for example, need to use other methods to work around the presence of STP, especially considering that OVS is more a full-featured virtual switch than the standard VMware vSwitch. In most cases, the workaround involves the use of link aggregation; specifically, the use of Link Aggregation Control Protocol (LACP), a standardized protocol that allows devices to automatically negotiate the configuration and use of link aggregates comprised of multiple physical links.

Now that you have the background, let’s dive into the details of how to make this work. These instructions on using LACP with OVS do make a few assumptions:

  1. First, I assume that OVS is already installed and working.
  2. I assume that the management traffic to/from your host is not running through OVS, and thus won’t be interrupted by any configurations you do here. If this is not the case, and you do have management traffic running through OVS, you might want to exercise some additional caution to ensure you don’t accidentally cut your connectivity to the host.
  3. I assume that you know how to configure your physical switch(es) to support LACP on the links coming in from OVS. The configuration will vary from switch vendor to switch vendor; refer to your vendor’s documentation for details.

This post was written using Ubuntu 12.04.1 LTS and Open vSwitch 1.4.0 (installed using apt-get directly from the Precise Pangolin repositories). The use of a different Linux distribution and/or a different version of OVS might make this process slightly different.

The first step is to add a bridge (substitute your desired bridge name for ovsbr1 in the following command):

ovs-vsctl add-br ovsbr1

Once the bridge is established, then you’ll need to create a bond. This is the actual link aggregate on OVS. The syntax for adding a bond looks something like this:

ovs-vsctl add-bond <bridge name> <bond name> <list of interfaces>

So, if you wanted to add a bond to ovsbr1 using physical interfaces eth1 and eth3, your command would look something like this:

ovs-vsctl add-bond ovsbr1 bond0 eth1 eth3

However, there’s a problem with this configuration: by default, LACP isn’t enabled on a bond. To fix this, you have two options.

  1. Change the command use to create the bond, so that LACP is enabled when the bond is created.
  2. Enable LACP after the bond is created.

For option #1, you’ll simply append lacp=active to the command to create the bond, like so:

ovs-vsctl add-bond ovsbr1 bond0 eth1 eth3 lacp=active

For option #2, you’d use ovs-vsctl set to modify the properties of the bond. Here’s an example:

ovs-vsctl set port bond0 lacp=active

Once the bond is created and LACP is enabled, you can check the configuration and/or status of the bond. Assuming that you’ve already configured your physical switch correctly, your bond should be working and passing traffic. You can use this command to see the status of the bond:

ovs-appctl bond/show <bond name>

The output from that command will look something like this:

bond_mode: balance-slb
bond-hash-algorithm: balance-slb
bond-hash-basis: 0
updelay: 0 ms
downdelay: 0 ms
next rebalance: 6415 ms
lacp_negotiated: true

slave eth4: enabled
    active slave
    may_enable: true

slave eth3: enabled
    may_enable: true

slave eth1: enabled
    may_enable: true

slave eth2: enabled
    may_enable: true

This command will show more detailed LACP-specific information:

ovs-appctl lacp/show <bond name>

This command returns a great deal of information; here’s a quick snippet:

---- bond0 ----
    status: active negotiated
    sys_id: 00:22:19:bd:db:dd
    sys_priority: 65534
    aggregation key: 4
    lacp_time: fast

slave: eth1: current attached
    port_id: 4
    port_priority: 65535

    actor sys_id: 00:22:19:bd:db:dd
    actor sys_priority: 65534
    actor port_id: 4
    actor port_priority: 65535
    actor key: 4
    actor state: activity timeout aggregation synchronized collecting
distributing partner sys_id: 00:12:f2:cc:6d:40 partner sys_priority: 1 partner port_id: 12 partner port_priority: 1 partner key: 10000 partner state: activity aggregation synchronized collecting
distributing

You can also use this command to view the configuration details of the bond:

ovs-vsctl list port bond0

The output from this command will look something like this:

_uuid               : ae7eb7ca-e3e0-4166-bcfb-4348071799e0
bond_downdelay      : 0
bond_fake_iface     : false
bond_mode           : []
bond_updelay        : 0
external_ids        : {}
fake_bridge         : false
interfaces          : [9963381b-6a7d-4a8f-acf8-86150361530e,
bee2df86-ed14-456b-8f3a-25fb00fa6040, daf5ac51-4135-4e3c-a937-c62dfc4b5e9f,
fcd2d6ef-9a18-452a-9a79-1c97e5a95ef2] lacp : active mac : [] name : "bond0" other_config : {lacp-time=fast} qos : [] statistics : {} status : {} tag : [] trunks : [] vlan_mode : []

In learning how to use LACP with OVS, I found this article to be extremely helpful.

If you have questions, or have additional information to share with me and/or other readers, please speak up in the comments. Thanks!

Tags: , , , ,

  1. TooMeeK’s avatar

    Well..
    yesterday I’ve deployed Open vSwitch on one of the servers, that has already configured bonding.
    I’ve just added vSwitch config to standard Ubuntu Server networking with bonding on LACP. So bond0 added to br0. And it works well.

    You may choose between ways to achieve same goal :)

  2. Eric’s avatar

    I implemented bonding according to your description.
    In my situation the bonding only seems to work for the native vlan.
    Tagged packet don’t seem to be accepted on the physical port. (Packets arive on other hosts on the network and the response is send.)
    I have this situation (CentOS 6.4):
    Physical interfaces eth0 and eth1, connected to a switch (Netgear GS724T). Ports configured as trunk.
    eth0 and eth1 added to bond0, like the above description.

    When I use bonding on Linux level. And add the linux bond to the openvswitch everything works fine.

    Do you know what I do wrong?

  3. KP’s avatar

    Thanks for the useful post, Scott. Regular Linux also supports bonding using ifenslave. I guess one can create a bond of the physical Nics using native linux and add the bond interface to OVS bridge instead of creating bond using OVS. Any difference between the two approaches? I assume native Linux run LACP. Does OVS run another separate LACP?

    And also, it will be interesting to see if Openstack allows us to specify bonded interfaces and hides the inner details.

  4. Atul Sharma’s avatar

    Question for the LACP veterans:

    How comprehensive is LACP support in OVS? Does it completely implement LACP as specified in IEEE Std 802.1AX-2008

  5. lock’s avatar

    hi, i just try to use your article to create LACP brige on two interfaces on ubuntu 12.04 LTS, all doned good until reboot, after reboot network interfaces that in bound of brige dosen`t start automatically, if i give some like “ifconfig eth1 up” bribge is come active… what can be wrong?

Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>