VLAN

You are currently browsing articles tagged VLAN.

FCoE and VLAN Trunking on Nexus 5000

Tuesday, November 3, 2009 in Networking, Storage by slowe | 2 comments

In my earlier post on how to configure FCoE on a Nexus 5000, one of the readers suggested in the comments that it was necessary to have the interfaces in VLAN trunk mode via the switchport mode trunk command. I didn’t pay that much attention to it because the interfaces were indeed in VLAN trunk mode.

Fast forward to yesterday, when I was troubleshooting a problem between a Gen2 QLogic CNA and the Nexus 5010 in my lab (I tweeted about it). Although the Ethernet side of the CNA works just fine, the CNA refuses to bring up an FCoE connection. In the process of troubleshooting, Brad Hedlund (check his outstanding web site) suggested to me in a Twitter direct message that I should double-check the VLAN trunking status of the interface. That part I’d already heard from the reader who commented on the first post, but the next part was new to me (emphasis mine):

Gen2 requires ’switchport mode trunk’ on the 5K. Gen1 doesn’t. Also make sure FCoE VLANs are allowed on the trunk.

Ah, now there’s something I hadn’t heard! That prompted me to do a bit of testing this morning (yes, I know I’m supposed to be studying for the VCDX Design Exam this afternoon). In my testing, I confirmed that a Gen1 CNA (I’m using Gen1 Emulex CNAs) does not require VLAN trunking to be enabled on the Ethernet interface.

There does appear to be a “gotcha” though: if the Ethernet interface is in access mode, it’s access VLAN must be the same as the FCoE VLAN; otherwise, the vfc interface will report down.

In summary:

  • If you are using a Gen2 CNA, you must put the Ethernet interface in VLAN trunk mode.
  • If you are using a Gen1 CNA, the Ethernet interface may be in either access mode or trunk mode.
  • If the interface is in trunk mode, be sure that you have allowed the FCoE VLAN via the switchport trunk allowed vlan command.
  • If the interface is in access mode, be sure that you have placed the interface in the FCoE VLAN via the switchport access vlan command.

If there are any other subtleties or nuances I’ve missed, please post them in the comments below so that future readers will benefit. Thank you!

Tags: , , , , ,

Why No Multi-Hop FCoE?

Tuesday, August 11, 2009 in Gestalt, Networking, Storage by slowe | 12 comments

Much ado has been made—some of it by yours truly—about the current lack of ability to create a multi-hop Fibre Channel over Ethernet (FCoE) fabric. After digging in deeper with Cisco during my recent Unified Computing System (UCS) class, I have some additional information to share about the different forms of multi-hop FCoE and why multi-hop FCoE still isn’t available.

Multi-hop FCoE falls into two basic scenarios:

  • FCoE initiators and/or FCoE targets separated from an FCF (fibre channel forwarder) by multiple hops through IEEE DCB-compliant Ethernet switches
  • Multiple FCFs chained together to connect FCoE initiators and FCoE targets

There are additional scenarios, but for now let’s discuss just these two.

In the first scenario, FCoE initiators and FCoE targets might be separated from an FCF by one or more IEEE DCB-compliant Ethernet switches (also known as an “FCoE passthrough”). In this situation, FCoE Initialization Protocol (FIP) would be required in order for the FCoE initiators and FCoE targets to communicate. Now that FIP support is beginning to emerge following the ratification of the FC-BB-5 standard in early June, this sort of scenario becomes more possible.

If you think like me (and if you do, I’m very sorry to hear that!), your next question is, “OK, what is an IEEE DCB-capable switch?” As it turns out, the Nexus 5000 can be an IEEE DCB-capable switch (or an FCoE passthrough). Cisco doesn’t advertise that fact because they don’t feel that building a solution out of a bunch of Nexus 5000 switches is the best approach. OK, fair enough, so the Nexus 5000 isn’t really designed to be used that way. So what other options are there? None of which I’m aware, at this point, so that makes it impossible to build multi-hop FCoE solutions today. When a valid IEEE DCB-capable switch or FCoE forwarder does appear then you’ll be able to build these sorts of designs—assuming that you have FIP support in both the FCoE initiators and the targets. (Note that you could mix pre-FIP components in here, but all such components would have to be connected directly to the FCF, and would only be able to communicate with other components connected directly to the FCF.)

In the second scenario, FCoE initiators and targets are connected directly to an FCF—like a Nexus 5000—but you’ve got multiple FCFs chained together to create a larger fabric. You might consider this analogous to linking multiple MDS 9000 series switches together with inter-switch links (ISLs). In this case, FIP support would still be necessary for initiators to connect to targets on a different FCF, but now there’s another wrinkle. You see, Cisco has the concept of a VSAN (think of it like a VLAN for Fibre Channel—this is a simplistic definition but reasonable enough to use). In the MDS world (keeping in mind that NX-OS, the software running on Nexus switches, has its roots in SAN-OS, the software that runs on MDS Fibre Channel switches), there is the concept of trunking E_ports, where multiple VSANs are carried on a single E_port between two MDS switches. Continuing the VSAN/VLAN analogy, a trunking E_port is analogous to an 802.1q VLAN trunk.

Bear with me, there’s a reason I’m telling you all this.

When you use FCoE on a Nexus 5000, you end up mapping each VSAN to a VLAN. When you need to move from one FCF to another FCF—i.e., from one Nexus 5000 to another Nexus 5000—how should the VSAN information be presented? Should the VSAN information reside in the 802.1q VLAN tag, so that an 802.1q VLAN trunk is considered a trunking E_port with regard to VSANs? Or should the VSAN information remain embedded in the FC commands that are encapsulated by Ethernet? This fundamental question has not yet been answered. There are advantages and disadvantages to each approach, and as the T.11 group responsible for FC-BB-5 and other FCoE standards hasn’t yet come to agreement yet on how to handle this, then it’s currently not possible to create the FCoE equivalent of trunking E_ports (I believe these will be referred to as VE_ports). Since you can’t create VE_ports, you can’t connect multiple FCFs together, and you can’t build a multi-hop FCoE fabric composed of multiple FCFs.

As you can see, then, that even with FIP present in all components, neither definition of multi-hop FCoE is possible today. Although a Nexus 5000 can function as an FCoE passthrough, Cisco doesn’t recommend that architecture. Without any other IEEE DCB-capable Ethernet switches available to use as an FCoE passthrough, that makes the first scenario impossible to build. Likewise, the inability to create VE_ports and trunk VSANs across multiple Nexus 5000 switches means that it’s impossible to build the second scenario today. While multi-hop FCoE is the ultimate goal, it’s just not possible right now.

Here’s some food for thought while you digest this information: how would a fabric extender change things? That’s a topic I’ll delve into in a future post, so stay tuned!

Of course, FCoE experts and wizards are encouraged to add your corrections, clarifications, and thoughts in the comments below.

Tags: , , , , ,

Using VMware ESX Virtual Switch Tagging with HP Virtual Connect

Monday, July 6, 2009 in Networking, Virtualization by slowe | 10 comments

In January 2008, SearchVMware.com published an article of mine titled VMware ESX Server Networking with HP Virtual Connect. In that article, I stated that one drawback of HP Virtual Connect was that it forced you to use External Switch Tagging (EST):

When used in conjunction with ESX Server, shared uplink sets force the use of EST because VLAN tags are stripped away by the Virtual Connect switch. Therefore, the ESX Server can’t use the VLAN tags, and must resort to a different vSwitch—each with one or more pNICs as uplinks—for each VLAN/associated network. This solution may be useful in some situations, but typically wouldn’t scale well for environments with many different VLANs.

It turns out that a firmware revision to the HP Virtual Connect software addresses this problem. As I’ve recently had the opportunity to work with an HP Virtual Connect Flex-10 module (you can expect to see some articles on Flex-10 in the near future), I wanted to revisit the idea of using HP Virtual Connect with VMware ESX/ESXi. In this post, I’ll describe how you go about configuring HP Virtual Connect so that you can use Virtual Switch Tagging (VST) and Shared Uplink Sets together. Each of the steps is described in one of the sections below.

Configure VC for VLAN Mapping

Before you even create the Shared Uplink Set, you must first configure the Virtual Connect module to use VLAN mapping instead of VLAN tunneling.

To access the option for configuring VLAN mapping, you can use the menu bar across the top of the right side of the HP Virtual Connect Manager. Simply click Configure > Ethernet Settings, then click on the Advanced Settings tab. There you will see the option to either tunnel VLAN tags or map VLAN tags. Choose to map VLAN tags. Optionally, you can click the check box to force server connections to use the same VLAN mappings; this will eliminate a step later but limits the overall flexibility of the solution. It’s up to you if you want to use this option.

Create the Shared Uplink Set

Now you’re ready to create the Shared Uplink Set. Using the menu bar across the top of the right side of the HP Virtual Connect Manager interface, simply choose Define > Shared Uplink Set. Specify a name for the uplink set, then choose the external uplink ports. While you here, you can also go ahead and create the associated networks you’ll need later.

Create the Associated Networks

Either while you’re creating the Shared Uplink Set or after the Shared Uplink Set has been created, you can add the Associated Networks. While you are editing the Shared Uplink Set, simply click the Add Network button under the Associated Networks area and define one or more VLANs. The following parameters are available when you define an Associated Network:

  • Network Name and VLAN ID: These are pretty self-explanatory. The VLAN ID here needs to match the external VLAN ID on the rest of the network.
  • Native: If this VLAN is marked as the native VLAN on the rest of your network, check this box. This network would then receive all of the untagged traffic on the uplink set.
  • Smart Link: If you would like this network to be marked as down if the uplinks go down, check this box.
  • Private network: With this box checked, the network will act like a private VLAN–nodes on this network cannot communicate with each other.
  • Advanced: This area allows you to set a custom speed for either the preferred speed or the maximum speed.

After you’ve defined the Associated Networks, then you’re ready to create the Server Profile—and that’s where the real magic in making VST is found.

Assign Networks in the Server Profile

Once again, you’ll use the menu bar across the right side of the HP Virtual Connect Manager to create a Server Profile. Simply select Define > Server Profile. In the Server Profile, you’ll add a network for each NIC present in the server blade (for a blade with Flex-10 NICs and a Flex-10 module, you will have eight NICs). When prompted for what network to associate to that NIC, choose Multiple Networks. Then click the small Edit button just to the right of the Network Name drop-down to show the Server VLAN to vNet Mappings screen.

The fact that you selected “Multiple Networks” when you added the connection to the Server Profile means that Virtual Connect will pass the VLAN tags up to the blade. The fact that you configured the Virtual Connect module to use VLAN mapping now means that you can create an association between the VLAN tags that a server uses and an corresponding Associated Network.

If you want to keep the server’s VLAN tags and the Associated Networks’ VLAN IDs matched up, just follow these steps:

  1. Check the box labeled Force Same VLAN Mappings As Shared Uplink Sets.
  2. Choose the Shared Uplink Set from the drop-down list.
  3. Place a check mark next to each Associated Network/vNet/VLAN. This tell the Virtual Connect module to include that VLAN.
  4. Place a check mark under Untagged for whichever Associated Network is should be handled as the untagged (native) VLAN.

If, on the other hand, you want to specify different server VLAN tags than Associated Network/vNet VLAN IDs, leave the check box for Force Same VLAN Mappings As Shared Uplink Sets unchecked, and specify the server VLAN ID that should be used for each Associated Network/vNet. For example, if the server was using VLAN ID 10 to refer to the Production network, but the Production network was using VLAN 1000 on the Associated Network and on the rest of the network, then choose the Associated Network that represents the Production network and specify a server VLAN ID of 10. This allows you to create a mapping between the VLAN tags the server uses and the VLAN tags the rest of the network uses.

After you’ve defined the Server Profile and created the vNet mappings, attach the Server Profile to a blade running VMware ESX/ESXi and you’re good to go! Within VMware ESX/ESXi, you would configure the vSwitches, distributed vSwitches, and port groups as you would normally.

How I Tested

My testing was performed on a HP Virtual Connect Flex-10 module running firmware revision 2.10. The Flex-10 module was uplinked via a single 10Gbps connection to an HP ProCurve switch. The blades were running VMware ESX 4.0.0.

Tags: , , , , ,

Changing vNIC Port Group Assignment in VMware with PowerShell

Monday, January 5, 2009 in Networking, Virtualization by slowe | 2 comments

Experienced PowerShell users won’t find this post very helpful, but less experienced PowerShell users—or even PowerShell newbies such as me—may find this handy. Today I had a need to change the port group assignment on the vNICs for a bunch of guest VMs in the lab. Rather than manually click through all these VMs just to change the port group, I decided to give PowerShell a try.

Thanks to this post by Cody Bunch and this Twitter response by Hal Rottenberg, I cobbled together this PowerShell command:

get-datacenter "Name" | get-vm | get-networkadapter | where-object { $_.networkname -like "OldPortGroup" } | set-networkadapter -networkname "NewPortGroup" -Confirm:$false

It worked like a champ! Obviously, you could limit the scope of this command by filtering the VMs that are returned with a wildcard pattern on the Get-VM command.

Thanks to Cody and Hal for their assistance!

Tags: , , , ,

VMware ESX Networking Articles

Friday, December 19, 2008 in Networking, Virtualization by slowe | 1 comment

Over the past few years—almost four years now that I’ve been writing here on this site—I’ve written quite a few articles on VMware ESX networking. Here’s a collection of links to some of the more notable, and hopefully more useful, articles in that group.

We’ll start with a classic article, but one that is still the number 1 post on this site. Written in December of 2006, this article on VMware ESX, NIC teaming, and VLAN trunking still generates lots of traffic even today. In that post, I describe how to use NIC teaming—more precisely, link aggregation or EtherChannel—along with VLAN trunking with VMware ESX. I guess this is a scenario that still causes problems with admins even today.

That first article was written with Cisco Catalyst switches in mind, but earlier this year I also published a version with information on VMware ESX, NIC teaming, and VLAN trunking with HP ProCurve switches as well.

One key piece to more fully understanding how VMware ESX interacts with VLAN tags is understanding what happens with untagged traffic, and this is something I described in this article on VMware ESX and the native VLAN. The key thing to remember here is to be sure to configure a VMware ESX port group to capture the untagged traffic, if that’s what you desire; otherwise, the traffic will just be ignored.

Similarly, a lot of VMware admins don’t fully understand the implications of the various load balancing policies on the vSwitch. The NIC teaming article from December 2006 assumed link aggregation (EtherChannel in the Cisco world), but how did VMware ESX behave in configurations without link aggregation? I tackled that subject in the post on NIC utilization in VMware ESX, and then provided a follow up post only a couple of months later. Both of these posts provide great information on how VMware ESX will place traffic onto the vSwitch uplinks.

Of course, if you find yourself needing to change the load balancing policy on a vSwitch, you might find this tip on using the CLI to change the vSwitch load balancing policy helpful, too. And this CLI guide to modify a port group may also prove useful.

In addition to an in-depth guide to NIC teaming, link aggregation, and VLAN trunking, I also provided a how-to on configuring VMware ESX, IP-based storage, and jumbo frames. While not officially supported by VMware, many users have reported success with this configuration and seen improved performance. (And before you ask: No, I haven’t run any performance comparisons myself yet.)

With the release of VI4—or whatever it ends up being called—there’ll be plenty of new networking fodder for me to write about: the Distributed vSwitch, the Cisco Nexus 1000V, and more. In the meantime, though, are there any other networking-oriented articles, guides, or how-to’s that readers would be interested in seeing published? I’m open to suggestions.

Tags: , , , , , , ,

More on VMware ESX NIC Utilization

Wednesday, October 8, 2008 in Networking, Virtualization by slowe | 6 comments

Earlier this year, I wrote an article about NIC utilization in VMware ESX in which I discussed some of the details around how VMware ESX uses NICs in various configurations. The testing that prompted that article was primarily centered around IP-based storage (software-based iSCSI and NFS), so naturally the discussion primarily centered around IP-based storage as well.

As a follow up to that article, I’ve been running some additional tests with a focus this time on the behavior of virtual machines (VMs) in configurations both with and without link aggregation. The results are very consistent with the findings from the previous set of tests, but with a few key items to keep in consideration when applied specifically to VMs.

Without Link Aggregation

As stated in the earlier article, using NIC teaming without link aggregation means the vSwitch is configured with one of these three load balancing settings:

  1. Route based on the originating virtual port ID
  2. Route based on source MAC hash
  3. Use explicit failover order

The primary focus here is “Route based on the originating virtual port ID,” since that’s the default vSwitch setting and the recommended setting from VMware when not using link aggregation.

Just as all the VMware docs explain, the tests show that a VM will use only one uplink, regardless of how many different connections are involved, and it will stay on that uplink until the uplink fails, at which time the VM will be moved to a different uplink according to the NIC failover order configured for that port group or vSwitch.

There are, however, a few other things to consider:

  • There is no guarantee that the VMs will be evenly distributed across the uplinks on a vSwitch or port group. Although I’ve seen references in the VMware documentation to indicate that VMs are balanced in some fashion (I could not find those references, however), real-world experience seems to indicate otherwise. In my tests, I had instances where four VMs were all “linked” (via their virtual port ID) to the same uplink.
  • It’s unclear at what point VMware ESX creates the link between the virtual port ID and the uplink. In my tests, I rebooted the guest OS and power cycled the VM only to have it come back on the same uplink again. Only a VMotion off the server and back again caused a VM to move to a new uplink. I’ve been trying to track down more information on the timing of the association between the virtual port ID and the uplink, but have thus far been unsuccessful.
  • There is no control over the placement of VMs onto uplinks without the use of multiple port groups. (Keep in mind that you can have multiple port groups corresponding to a single VLAN.)
  • Because multiple VMs could be assigned to the same uplink, and because users have no control over the placement of VMs onto uplinks, it’s quite possible for multiple VMs to be assigned to the same uplink, or for VMs to distributed unevenly across the uplinks. Organizations that will have multiple “network heavy hitters” on the same host and vSwitch may run into situations where those systems are all sharing the same network bandwidth.

These considerations are not significant, but they are not insignificant, either. The workaround for the potential problems outlined above involves using multiple port groups with different NIC failover orders so as to have more fine-grained control over the placement of VMs on uplinks. In larger environments, however, this quickly becomes unwieldy. The final release of the Distributed vSwitch will help ease configuration management in this sort of situation, but that’s still out in the future yet.

VM Networking with Link Aggregation

Using NIC teaming with link aggregation means that the vSwitch is configured with “Route based on ip hash” and that the physical switch has been configured for EtherChannel or static 802.3ad/LACP.

<aside>By the way, static 802.3ad/LACP in the Cisco IOS world means the use of the “channel-group X mode on” command as opposed to “channel-group X mode active”; the first command uses static link aggregation whereas the second uses dynamic link aggregation. Static is supported; dynamic is not.</aside>

In this environment, the tests showed exactly what we expected; traffic from VMs was distributed across the uplinks in a dynamic fashion based upon the source and destination IP addresses. While some of these things may be common sense, there are a few things to consider:

  • Depending upon the number of uplinks, it’s certainly possible that some traffic streams may end up getting hashed onto the same uplink. One a traffic stream is placed onto an uplink, it won’t move off that uplink until the flow is complete.
  • The way in which outbound traffic from the VMs will be placed onto the uplinks won’t necessarily be the same as the way the physical switch places inbound traffic onto the uplinks. A good resource on how Cisco handles placing traffic onto members of an EtherChannel bundle is this page.
  • Multiple connections to or from a VM might utilize multiple uplinks but aren’t necessarily guaranteed to use multiple uplinks.

Again, some of the points above are simple common sense, but they should be kept in mind nevertheless. If the workloads that are being hosted on VMware ESX are such that having more aggregate bandwidth would be beneficial, then using link aggregation is really the only way to do it. I suppose it would be possible to use NIC teaming inside the guest OS, assuming the VM was configured to use e1000 NICs, but I’ve never tested this configuration. (Anyone else tested it?)

Tags: , , , , , ,

VLANs on ESX with Nortel Switches

Tuesday, June 24, 2008 in Networking, Virtualization by slowe | 8 comments

I ran into a recent issue with a customer who was having problems getting VLANs to work as expected with ESX. The basic scenario was that ESX would refuse to work properly with a VLAN that was marked as the native (or untagged) VLAN. This was causing no end of grief for this customer.

I’ve discussed VLANs extensively—first with this blog post, then again here, and again in this SearchVMware.com tip—so I was confident that I could help the customer resolve this issue. Granted, the customer was using Nortel switches, with which I am completely unfamiliar, but a switch is a switch, right?

Not quite. While the configuration seemed correct in all ways, it turns out there is a checkbox somewhere labeled “untag-default-vlan”. If this box is not checked, then the default VLAN gets tagged. Since ESX wasn’t configured with a VLAN tag, then it doesn’t see the network traffic. Once that box gets checked, then the default (or native) VLAN doesn’t get tagged and will be properly recognized by an ESX port group without a VLAN tag configured.

So, if you’re using Nortel switches and having problems with VLANs, double-check this setting.

Tags: , , , ,

Configuration for Protecting VMotion

Friday, March 7, 2008 in Networking, Security, Virtualization by slowe | No comments

As a follow-up to my earlier post about using VLANs with VMotion, I wanted to also share a brief configuration snippet (based on a Cisco IOS-based switch) that aligns with the recommendations in that article. This is nothing new to those who are familiar with IOS, but for those readers who are not it may provide some helpful information.

For the purposes of this configuration, I’m making the following assumptions:

  • VLANs 100 and 200 are the VMotion VLAN and some other production VLAN, respectively (perhaps a VLAN carrying virtual machine traffic, or a VLAN carrying Service Console traffic)
  • VLAN 4094 is the “ESX Trunk” VLAN, which isn’t used anywhere else in the network for any purpose

Here’s the recommended port configuration for ports connecting to an ESX Server:

interface g0/1
switchport trunk encapsulation dot1q
switchport trunk native vlan 4094
switchport mode trunk
switchport trunk allowed vlan 100,200,4094
switchport noneg
spanning-tree portfast-trunk

Technically, the “switchport noneg” command won’t really do anything; it disables DTP (Dynamic Trunking Protocol) but DTP isn’t supported by ESX Server anyway.

A couple of notes about this configuration:

  • Refer back to my article on the native VLAN with ESX Server; by setting the native VLAN to 4094 (the “ESX Trunk” VLAN), you won’t carry that traffic into the ESX Server. If used on a vSwitch that also carried Service Console traffic, then it could impact automated build scripts.
  • If you needed to combine this configuration with EtherChannel, refer back to my original article on ESX Server, NIC teaming, and VLAN trunking.

Keep in mind the other recommendations as well: explicitly control trunking to other devices and explicitly control the transmission of VLANs across trunks to control “VLAN leakage”.

CCIEs and other experts, I’d welcome any other suggestions as well as recommended commands to use in the switch configuration to help maximize security and minimize exposure to VLAN-based attacks.

Tags: , , , , , , , ,

VMotion and VLAN Security

Wednesday, March 5, 2008 in Networking, Security, Virtualization by slowe | 8 comments

Xensploit, as it’s called, is the recently demonstrated exploit that allows virtual machines (VMs) that are “in flight” during a live migration (XenMotion in Citrix XenServer, VMotion in VMware ESX Server) to be manipulated. If you haven’t yet read the PDF that describes Xensploit, I highly encourage that you take a look at it. It’s very enlightening as to exactly what can be done to an in-flight VM.

Naturally, the best way to protect against this particular problem is to guard the integrity of the live migration network. For simplicity’s sake, I’ll refer to this as the VMotion network from this point on, but keep in mind that it is equally applicable to any network connections on any virtualization platform that uses live migration.

The most surefire way to protect the VMotion network is to place it on its own dedicated, physically separate network, using separate physical NICs plugged into separate physical switches that do not possess any connections to production networks. This will ensure that unauthorized access to the VMotion network is prevented, but comes with disadvantages as well: this configuration requires more physical NICs and more physical switches than other configurations.

In implementations with limited numbers of physical NICs, however, this isn’t really an option. In these cases, the use of Layer 2 VLANs and multiple port groups on a single ESX Server vSwitch to allow VMotion traffic to share the same physical NICs and the same physical switches as other traffic is a very common solution. In fact, it’s a solution that I’ve recommended many, many times. But does this configuration provide enough protection for the VMotion network?

The real question is, does a simple Layer 2 VLAN offer enough protection? That question, in turn, spawns other questions: what kinds of attacks are there against Layer 2 VLANs? Is it possible for traffic to hop across VLAN boundaries?

Armed with those questions, I set out to do some research. You can see some links in my del.icio.us bookmarks that pertain to the research I did. Basically, I found that Layer 2 VLAN attacks boil down to two basic types:

  1. The first type involves a malicious host pretending to be a switch and forming a 802.1Q VLAN trunk with the real switch, which then passes traffic from all VLANs across to the malicious host.
  2. The second type involves double-tagged 802.1Q frames and the native VLAN, whereby traffic can, under specific circumstances, hop from one VLAN to another without any Layer 3 routing involved.

(Network and security gurus out there feel free to elaborate or correct me on this information.)

The best way to address attack vector #1 is to explicitly disable automatic trunk negotiation on all ports that don’t need to be trunks. From my research and my (relatively) limited knowledge of Cisco IOS, this command should do it:

switchport mode access

This explicitly forces the switchport into a state where it will not negotiate an 802.1Q VLAN trunk with another device, hence killing attack vector #1 dead in its tracks. Again, this should only be done on the ports that are not connected to the ESX Servers; otherwise, you’re shooting yourself in the foot. Keep in mind that uplinks to other switches, ports going out to IP phones, etc., may also need to be configured as VLAN trunks. Really, the issue is about controlling the creation of unauthorized VLAN trunks in order to control VLAN leakage. One of the CCIEs at the office mentioned a “switchport noneg” command, but I’m not familiar with that one; anyone have more details?

Addressing attack vector #2 is also relatively straightforward. Since the VLAN hopping exploit takes advantage of the nature of the native VLAN (which I’ve discussed before here), setting the native VLAN on the trunk ports connecting to the ESX Servers to a VLAN that is not used anywhere else in the network will prevent VLAN hopping. For example:

switchport trunk native vlan 4094

From my NIC teaming and VLAN trunking article (one of the most popular articles on the site, by the way), you’ll see that I recommended at that time the creation of a VLAN that is used only as the native VLAN for 802.1Q trunks. At that time, I didn’t fully understand why; now, after additional research, I understand why I needed to do that and I also recognize that the suggested configuration provides a layer of protection against VLAN hopping attacks.

To summarize:

  • To protect against malicious hosts forming unauthorized 802.1Q trunks, disable automatic trunk negotiation and explicitly/manually create trunks. The key here is to ensure that VLANs don’t inadvertently “leak” beyond where they should (also see note below about specifying the allowed VLANs).
  • To protect against VLAN hopping, create a VLAN that is used only as the native VLAN on the 802.1Q trunks connecting to the ESX Servers. This VLAN must not be used anywhere else in the LAN. Set this VLAN as the native VLAN on the trunks into the ESX Servers.

In addition, my networking mentors also recommended the “switchport trunk allowed vlan” command to specify which VLANs are allowed to cross 802.1Q VLAN trunks. This will help ensure that VMotion traffic is limited to only those switches that absolutely must carry it; again, we’re seeking to control VLAN leakage.

With these configurations in place, using a Layer 2 VLAN to carry VMotion traffic on the same physical NICs and same physical switches as other types of traffic is fairly well-protected against malicious interference. While it is not as secure as a physical separate, dedicated network, it is secure enough for most organizations and the reduction in infrastructure needs generally outweighs the risks.

More information and discussion about Xensploit—and protecting against Xensplot—at the following links:

Keeping Your VMotion Traffic Secure
Two vulnerabilities found in VMware virtualization products
‘Live’ VMs at Risk While in Transit
News Flash: If You Don’t Follow Suggested Security Hardening Guidelines, Bad Things Can Happen…

UPDATE: I’ve updated the wording above to more properly reflect the goal behind the use of the “switchport mode access” command, and when it should be used. Colin, thanks for the feedback and clarification!

Tags: , , , , , ,

HP VirtualConnect Clarification

Wednesday, January 30, 2008 in Networking, Virtualization by slowe | 4 comments

SearchVMware.com recently published an article of mine about using HP VirtualConnect with ESX Server and the impact it has on configuring ESX Server networking.  Based on the comments to my original post about the article, it appears that I didn’t adequately explain the interaction between VirtualConnect’s Standard Ethernet Networks and VLAN tagging.  I’d like to thank those readers that asked about this issue and take a moment to clarify.

Standard Ethernet Networks are defined in the HP VirtualConnect Manager and allow you to control the mapping between physical NIC ports on the blades (the “downstream ports”) and the connections from the VirtualConnect switches to the external network infrastructure (the “upstream ports”).  For example, I could define a Standard Ethernet Network called “Production” and specify that Production would uplink either via Slot 2 Port 7 or Slot 2 Port 8.

That part is pretty straightforward.  Here’s how it plays into VLAN tagging with ESX Server.  Most organizations prefer to use VST (Virtual Switch Tagging; described in more detail in this article).  To use VST, the ports on the network infrastructure must be configured as 802.1Q VLAN trunks so that the external physical switches will pass the VLAN tags to the vSwitches inside ESX Server, where ESX Server can then handle the traffic appropriately.

To use VST with VirtualConnect and Standard Ethernet Networks, there is no difference.  The VirtualConnect upstream ports, as defined in the Standard Ethernet Network configuration, must be plugged into physical switch ports that are configured as 802.1Q VLAN trunks.  When you plug the upstream port into an 802.1Q VLAN trunk, the downstream port—the port going to the ESX Server—automatically becomes an 802.1Q VLAN trunk as well.  The VLAN tags will pass all the way to the ESX Server’s vSwitches, where ESX Server can deal with the traffic appropriately.

Likewise, if the VirtualConnect’s upstream ports are plugged into a static access port—a port which is not configured as an 802.1Q VLAN trunk but instead carries traffic only for a single VLAN—then the downstream ports also become static access ports and you are, effectively, using External Switch Tagging (EST).

I stated this in the original article in the third paragraph in the section titled “How Virtual Connect differs in ESX”:

In either way, these Ethernet networks will “pass through” the 802.1Q status of the physical switch port to which it is uplinked. If the physical switch port to which they are connected is configured as an 802.1Q VLAN trunk, then the downstream ports will act as 802.1Q VLAN trunks. Likewise, if the uplink is connected to a switch port that is configured as a static access port, then the downstream ports will act as static access ports.

Shared Uplink Sets, on the other hand, are different.  They don’t behave in the same way as Standard Ethernet Networks.  With Shared Uplink Sets, you are forced to use EST because the VLAN tags are stripped away at the VirtualConnect level.  The associated networks that are defined in VirtualConnect Manager define the different VLANs, and each downstream port is connected to an associated network.  Unlike with Standard Ethernet Networks, no VLAN tags are passed up to the ESX Server with Shared Uplink Sets.

So, to summarize:

  • Standard Ethernet Network uplink connected to external switch port configured as 802.1Q VLAN trunk allows ESX Server to see VLAN tags and supports both VST and Virtual Guest Tagging (VGT)
  • Standard Ethernet Network uplink connected to external switch port configured as static access port only allows EST configuration
  • Shared Uplink Set only allows EST configuration

I hope this clarifies the interaction between HP VirtualConnect and ESX Server networking.  I welcome any further questions or clarifications in the comments below.  Thanks!

Tags: , , , ,

« Older entries