blog.scottlowe.org

A colleague at work just turned me on to the news that Brocade will be buying Foundry for $3 billion. This sets the stage for Brocade to compete even more directly with Cisco, not only in storage networking but now also in Ethernet networking.

Thanks for the heads-up, Greg!

In early December of 2006, I wrote a very popular article on VMware ESX, NIC teaming, and VLAN trunking. In that article, I laid out the configuration for using both NIC teaming and VLAN trunking. In particular, the NIC teaming configuration in that article described the use of Cisco Gigabit EtherChannel for link aggregation, in which both the physical switch and the vSwitch are configured to distribute traffic across all the links between them.

Since that time, the question has come up many times: which method is better, with EtherChannel or without? Many engineers prefer not to use EtherChannel (or its standardized equivalent, static LACP/802.3ad) because of the added complexity involved. It’s easier to just team the NICs at the vSwitch level and leave the physical switches alone. That is true, but what about performance? And what impact does this have on NIC utilization?

There are two ways of handling NIC teaming in VMware ESX:

1. Without any physical switch configuration
2. With physical switch configuration (EtherChannel, static LACP/802.3ad, or its equivalent)

In the NIC teaming/VLAN trunking article I referenced above, I noted that there is a corresponding vSwitch configuration that matches each of these types of NIC teaming:

1. For NIC teaming without physical switch configuration, the vSwitch must be set to either “Route based on originating virtual port ID” or “Use explicit failover order”
2. For NIC teaming with physical switch configuration—EtherChannel, static LACP/802.3ad, or its equivalent—the vSwitch must be set to either “Route based on ip hash” or “Route based on source MAC hash”. The majority of the testing I’ve done is with “Route based on ip hash,” so that’s where I’ll focus most of this discussion. Either way, the physical switch’s link aggregation mechanism should match whatever configuration is set on the vSwitch.

In order to better understand how these settings and different configurations affect NIC utilization, I set out to do some tests in the lab. Most of my tests were centered around IP-based storage from the host (i.e., using NFS or iSCSI for VMDKs), and only tested two basic configurations: using “Route based on originating virtual port ID” and no link aggregation and using “Route based on ip hash” with link aggregation. Although the tests were slanted toward IP-based storage traffic, the underlying principles should be the same for other types of traffic as well. Here’s what I found.

NIC Teaming Without Link Aggregation

First, it’s important to understand the basic behavior in this configuration. Because the vSwitch is set to “Route based on originating virtual port ID”, network traffic will be placed onto a specific uplink and won’t use any other uplinks until that uplink fails. (This is described in more detail in this PDF from VMware.) Every VM and every VMkernel port gets its own virtual port ID. These virtual port IDs are visible using esxtop (launch esxtop, then press “n” to switch to network statistics). That’s simple enough, but what does this mean in practical terms?

• Each VM will only use a single network uplink, regardless of how many different connections that particular VM may be handling. All traffic to and from that VM will be place on that single uplink, regardless of how many uplinks are configured on the vSwitch.
• Each VMkernel NIC will only use a single network uplink. This is true both for VMotion as well as IP-based storage traffic, and is true regardless of how many uplinks are configured on the vSwitch.
• Even when the traffic patterns are such that using multiple uplinks would be helpful—for example, when a VM is copying data to or from two different network locations at the same time, or when a VMkernel NIC is accessing two different iSCSI targets—only a single uplink will be utilized.

This last bullet is particularly important. Consider the implications in a VMware Infrastructure 3 (VI3) environment using the software iSCSI initiator with multiple iSCSI targets. Even though multiple iSCSI targets may be configured, all the iSCSI targets will share one uplink from that vSwitch using this configuration. Obviously, that is not ideal.

Note that this doesn’t really impact VMotion traffic, since VMotion is a point-to-point type of connection. VMotion would only be impacted if placed on a vSwitch with other types of traffic and their virtual port IDs were assigned to the same uplink.

NIC Teaming With Link Aggregation

In this configuration, EtherChannel/static LACP/802.3ad is configured on the physical switch and the ESX vSwitch is configured for “Route based on ip hash.” With this configuration, the behavior changes quite dramatically.

• Traffic to or from a VM could be placed onto any uplink on the vSwitch, depending upon the source and destination IP addresses. Each pair of source and destination IP addresses could be placed on different uplinks, but any given pair of IP addresses can use only a single uplink. In other words, multiple connections to or from the VM will benefit, but each individual connection can only utilize a single link.
• Each VMkernel NIC will utilize multiple uplinks only if multiple destination IP addresses are involved. Conceivably, you could also use multiple VMkernel NICs with multiple source IP addresses, but I haven’t tested that configuration.
• Traffic that is primarily point-to-point won’t see any major benefit from this configuration. A single VM being accessed by another single client won’t see a traffic boost other than that possibly gained by the placement of other traffic onto other uplinks.

This configuration can help improve uplink utilization on a vSwitch because traffic is dynamically placed onto the uplinks based on the source and destination IP addresses. This helps improve overall NIC utilization when there are multiple VMs or when a VMkernel NIC is accessing multiple IP-based storage targets. Note again, though, that individual connections will only ever be able to utilize a single uplink.

Practical Application

I come back again to the question I asked earlier: what does this mean in practical terms?

• If you want to scale IP-based storage traffic, you’ll have to use link aggregation and multiple targets. Using link aggregation with a single target (destination IP address) won’t use more than a single uplink; similarly, no link aggregation with multiple targets will still result in only a single uplink being used. Only with link aggregation and multiple targets will multiple uplinks get used.
• Link aggregation will help with better overall uplink utilization for vSwitches hosting VMs. Because there are multiple source/destination address pairs in play, the vSwitch will spread them around the uplinks dynamically.
• To achieve the best possible uplink utilization as well as provide redundancy, you’ll need physical switches that support cross-stack link aggregation. I believe the Cisco Catalyst 3750 switches do, as do the Catalyst 3120 switches for the HP c-Class blade chassis. I don’t know about other vendors, since I deal primarily with Cisco.

Clearly, this has some implications for efficient and scalable VI3 designs. I’d love to hear everyone’s feedback on this matter. In my humble opinion, extended conversations about this topic can only serve to better educate the community as a whole.

My Virtualization Short Take series seems to be reasonably popular, so I thought I’d expand into another area that interests me: storage. Here is Storage Short Take #1! If this proves helpful or useful to readers, I’ll continue the series on an irregular basis.

• If you’re new to SnapMirror, especially synchronous SnapMirror, this synchronous SnapMirror configuration guide may prove very helpful to you.
• Good friend Nick Triantos discusses NetApp’s Storage Recovery Adapter (SRA) for VMware Site Recovery Manager (SRM). While the discussion is specific to NetApp, it’s a good example of how the storage vendors are responsible for implementing vendor-specific functionality in the SRA. The other storage vendors supporting SRM are responsible for the same things for their storage arrays and storage array functionality.
• Isn’t this the truth—everyone and their brother has “storage virtualization” functionality built into their products these days. Frankly, I’m tired of hearing about it.
• If you’re running VMs on NFS on NetApp storage, you’ll want to note this knowledge base article (NOW login required). It notes that a SCSI disk timeout increase may need to be set in order to accommodate cluster failover times.
• I recently came across this white paper from Emulex and Cisco regarding the use of N_Port ID Virtualization (NPIV) in a VMware environment. Personally, I found it to be a bit light on the technical details and a bit heavy on the marketing side, but otherwise useful.
• This tool from NetApp (NOW login required) can help with approximating storage I/O. It’s not perfect, but it might help provide some rough estimates. I’m sure other vendors have similar tools; readers are encouraged to share links to those tools in the comments.

Well, that’s it for now. I’d love to hear feedback (good or bad) from readers as to whether this is even remotely useful or interesting.

Here’s Virtualization Short Take #12, a collection of links I’ve gathered over the last week or so and my thoughts on them. Enjoy!

• For those that missed it in the Release Notes, VMware added support for Storage VMotion and 10Gb Ethernet with iSCSI SANs, as outlined in this VI Team blog entry. I went back and reviewed the Release Notes and didn’t see this listed anywhere, so this is news to me. Of course, I already knew that Storage VMotion worked just fine with iSCSI, but this added formal support for iSCSI.
• Virtualfuture.info published some good recommendations for running Citrix in a VI3 environment. If you run Citrix Presentation Server…er, XenApp…in a VI3 environment, these tuning tips may prove quite handy.
• VMware’s Virtual Reality blog posted an entry on some of the architectural advantages of VMware Infrastructure in comparison to the two leading competitors, Xen (any Xen-based solution) and Hyper-V. Many of the things listed as advantages by VMware are severe points of contention with the other vendors, such as the direct vs. indirect I/O model. Ultimately, time will tell which model was the best; I honestly don’t know enough about the deep dark internals to really state which is better. One thing I am glad to see pointed out is the true comparison of hypervisor sizes; Microsoft can say all they want that Hyper-V is only 600K in size and therefore is the “thinnest” hypervisor, but the truth of the matter is that Hyper-V can’t run without Windows Server 2008 in the parent partition. As a result, it doesn’t really matter how “thin” Hyper-V is, does it?
• Via Mike Laverick, I learned that Microsoft may have brought up the whole 64-bit hypervisor vs. 32-bit hypervisor argument yet again. Mike used a snippet from this Microsoft Virtualization Team Blog entry; in reading it myself, I don’t get quite the same 64-bit vs. 32-bit that Mike picked up. That’s good, because I didn’t want to have to go there again. Personally, the tone I picked up from the whole article was one of educating people far too accustomed to Virtual Server/VirtualPC and trying to educate them on how Hyper-V is different.
• Virtualization analyst Chris Wolf recently posted an entry in which he questioned if Apple would capitalize on the opportunity that virtualization is creating. It’s an interesting scenario, one that is similar to a scenario that I discussed a couple of years ago in a piece titled “Application Agnosticism.” In that article, I suggested that seamless host-guest interactions with virtualization software (now implemented by VMware as Unity and by Parallels as Coherence) would usher in a new wave of computing. I suggested that Mac OS X was ahead of the curve because of its ability to run native OS X applications, UNIX applications, X11 applications, Windows applications via WINE (or the commercial variant CrossOver Office), and applications from any other operating system via virtualization. Sounds like I may have been a bit ahead of my time!
• Chad continues discussing VMware HA with another post on some additional configuration options for HA. Also check out the comments with links to even more information on HA’s advanced configuration options.
• This VMware KB article has some good information on getting LUN identification information. The breakdown of the command-line output from esxcfg-mpath is particularly helpful (and for that reason I’ve added it to my del.icio.us bookmarks).
• Rich of VM /ETC shares with us a “Doh!” moment he had when he saw this simple method for identifying VMs with snapshots. Sometimes it’s the simplest solutions that evade us the longest. Here’s what I want to know: Aaron, what exactly does “/HEADDESK” mean, anyway?
• This article at SearchNetworking.com brings to light some of the challenges networking professionals face with server virtualization. I do agree with one point made in the article regarding the mapping of applications—what the end users really care about—to the networking infrastructure. VMware’s support for CDP in recent versions of VMware Infrastructure is a step in the right direction, but there is still more work to do for sure. I’m not so sure about the rest of the points in the article, but I may be an exception to the norm; I was a CCNA for a while (on track for CCNP) and have done my fair share of Cisco configurations, so I’m no stranger to the networking world. The use of VLANs to ease configuration in a server virtualization environment seems just second nature to me. Also, I did note that the author indicated that “server administrators sometimes inappropriately configure the switches to create a loop” (referring to vSwitches in ESX). How exactly does that happen? I’ve never seen a way to link two vSwitches together without using a VM.

As always, readers’ thoughts are welcome in the comments!

Back on March 5 blogger extraordinaire Alessandro Perilli of virtualization.info revealed that Cisco had chosen KVM as the virtualization platform for IOS-XE, the new Linux-based version of IOS that runs on the recently introduced ASR series of routers.

If you were like me, you may have been wondering exactly how Cisco was putting KVM to use. No need to wonder any longer! Colleague and fellow blogger Colin McNamara has written up a detailed and in-depth discussion of the ASR1000 and how it uses KVM to provide virtualized instances of IOS-XE. Colin also discusses the role of the QuantumFlow processor and, believe it or not, the role of Popeye and Spinach. (Go read the article. It will make sense when you’re done.) Nice work, Colin!

This is a handy little trick that many of you may already know. Starting with version 3.5, VMware added support for Cisco Discovery Protocol (CDP) on the ESX Server vSwitches. CDP support is enabled on a vSwitch with this command:

esxcfg-vswitch -B both vSwitch0

The “-B” parameter is case-sensitive; the “-b” (note the lowercase B) displays CDP status while the “-B” (uppercase B) configures CDP.

Once CDP support is enabled on the vSwitch—and assuming it is enabled on the physical switch—running the “show cdp neighbor” IOS command will show the link between each physical switch port and the matching ESX Server NIC. The output will look something like this:

Capability Codes: R-Router, T-Trans Bridge, B-Source Route Bridge
                  S-Switch, H-Host, I-IGMP, r-Repeater, P-Phone

Device ID Local Intrfce Holdtme Capability Platform      Port ID
s3        Gig 0/26        147      T S     WS-C3524-XFas 0/24
esx04     Gig 0/22        168       S      VMware        ESXvmnic0
esx04     Gig 0/21        168       S      VMware        ESXvmnic1

As you can see in the output above, the CDP output clearly links the physical switch port and the ESX Server NIC. This makes it incredibly easy to identify the NICs in the server. This is particularly helpful in blade situations, since you can’t exactly unplug the NIC and see which one goes down with “esxcfg-nics -l” (a common approach to identifying the NICs in the server). Of course, this requires Cisco switches in the blade chassis. Since the internal port mappings on the blade chassis determine which NICs connect to which ports, this command adds the mapping within ESX Server and lets us quickly and definitively identify the NICs in the server as seen by ESX Server.

As a follow-up to my earlier post about using VLANs with VMotion, I wanted to also share a brief configuration snippet (based on a Cisco IOS-based switch) that aligns with the recommendations in that article. This is nothing new to those who are familiar with IOS, but for those readers who are not it may provide some helpful information.

For the purposes of this configuration, I’m making the following assumptions:

• VLANs 100 and 200 are the VMotion VLAN and some other production VLAN, respectively (perhaps a VLAN carrying virtual machine traffic, or a VLAN carrying Service Console traffic)
• VLAN 4094 is the “ESX Trunk” VLAN, which isn’t used anywhere else in the network for any purpose

Here’s the recommended port configuration for ports connecting to an ESX Server:

interface g0/1
switchport trunk encapsulation dot1q
switchport trunk native vlan 4094
switchport mode trunk
switchport trunk allowed vlan 100,200,4094
switchport noneg
spanning-tree portfast-trunk

Technically, the “switchport noneg” command won’t really do anything; it disables DTP (Dynamic Trunking Protocol) but DTP isn’t supported by ESX Server anyway.

A couple of notes about this configuration:

• Refer back to my article on the native VLAN with ESX Server; by setting the native VLAN to 4094 (the “ESX Trunk” VLAN), you won’t carry that traffic into the ESX Server. If used on a vSwitch that also carried Service Console traffic, then it could impact automated build scripts.
• If you needed to combine this configuration with EtherChannel, refer back to my original article on ESX Server, NIC teaming, and VLAN trunking.

Keep in mind the other recommendations as well: explicitly control trunking to other devices and explicitly control the transmission of VLANs across trunks to control “VLAN leakage”.

CCIEs and other experts, I’d welcome any other suggestions as well as recommended commands to use in the switch configuration to help maximize security and minimize exposure to VLAN-based attacks.

I discovered another ePlus blogger today.  Last week I mentioned that fellow ePlus engineer Aaron Delp has launched his site, The Blade Blog (now called BladeVault.info); today I discovered that Colin McNamara also runs his own blog as well.  Colin is a CCIE whose blog, quite naturally, focuses on networking and Cisco equipment.  Most of his technical stuff is, frankly, over my head; I only know enough to get myself into trouble.

Colin’s RSS feed is here.

UPDATE: I’ve updated Aaron’s URL above to reflect the site’s new location.

In my previous article about using NetApp multi-mode VIFs with Cisco switches, I mentioned that you could—at that time—only use 802.3ad static link aggregation:

Be aware that Data ONTAP’s multi-mode VIFs are only compatible with static 802.3ad link aggregation; you can’t use PAgP (Cisco proprietary protocol).  I would assume dynamic LACP is also incompatible.  For this reason we used the “channel-group 1 mode on” statement instead of something like “channel-group 1 mode desirable”.

I recently got some feedback from a NetApp SE in my area; this SE informed me that Link Aggregation Control Protocol (LACP, part of the IEEE 802.3ad specification) is indeed supported with Data ONTAP version 7.2.  This KB article on the NetApp NOW site (login required) indicates that ONTAP 7.2.1 is required in order to use a LACP VIF.

There are a couple important requirements to note; these are laid out in the referenced KB article:

1. Dynamic multimode VIFs should use IP address-based load balancing.  This means that the Cisco switch or the channel group must also use IP address-based load balancing.
2. Dynamic multimode VIFs must be first-level VIFs.  This makes sense; LACP is a Layer 2 protocol, so layering a LACP VIF on top of other VIFs just doesn’t work.

To create the dynamic multimode VIF on the Data ONTAP side, the command is pretty simple:

vif create lacp <vif name> -b ip {interface list}

On the Cisco side, the commands are very similar:

s3(config)#int port-channel1
s3(config-if)#description LACP multimode VIF for netapp1
s3(config-if)#int gi0/23
s3(config-if)#channel-protocol lacp
s3(config-if)#channel-group 1 mode active

These commands would be repeated for all physical ports that should be included in the LACP bundle.  Note the differences from the earlier commands in the previous article; here we use “channel-group 1 mode active” instead of “channel-group 1 mode on”.  We also added the “channel-protocol lacp” command.

Together, these commands will establish a LACP-based link aggregate between a NetApp storage system running Data ONTAP version 7.2.1 or higher and a Cisco IOS-based switch.

Thanks to Jeff, our NetApp SE, for providing the updated information.

I just saw this headline from virtualization.info about Cisco being the first to announce a third-party virtual switch for ESX Server. Honestly, I’m not too terribly surprised.

Surely, the announcement of John Chambers as a keynote speaker at VMworld 2007, followed by the investment of $150 million by Cisco into VMware surely had to signal that something was going on.  This revelation that Cisco plans on being the first to announce a third-party virtual switch for ESX Server is honestly not a surprise.  In fact, if I recall correctly, there were rumors of this last year at VMworld.

Still, despite the fact that it’s not really a surprise, I’m still excited about the possibility that this is actually the case.  I’m not a Cisco expert, but I’d love to have the ability to run Cisco-based switches on ESX Server (assuming the overhead isn’t too significant).  That should make integration of ESX Server with Cisco-based networks much easier than it has been in the past.  What’s next—SAN switches on ESX?  (I may not be too far off, there, with all the rumors and discussions about NPIV support in ESX Server.)

UPDATE:  Ryan Glover over at p2vd.com shares his thoughts on the announcement here.