In January 2008, SearchVMware.com published an article of mine titled VMware ESX Server Networking with HP Virtual Connect. In that article, I stated that one drawback of HP Virtual Connect was that it forced you to use External Switch Tagging (EST):
When used in conjunction with ESX Server, shared uplink sets force the use of EST because VLAN tags are stripped away by the Virtual Connect switch. Therefore, the ESX Server can’t use the VLAN tags, and must resort to a different vSwitch—each with one or more pNICs as uplinks—for each VLAN/associated network. This solution may be useful in some situations, but typically wouldn’t scale well for environments with many different VLANs.
It turns out that a firmware revision to the HP Virtual Connect software addresses this problem. As I’ve recently had the opportunity to work with an HP Virtual Connect Flex-10 module (you can expect to see some articles on Flex-10 in the near future), I wanted to revisit the idea of using HP Virtual Connect with VMware ESX/ESXi. In this post, I’ll describe how you go about configuring HP Virtual Connect so that you can use Virtual Switch Tagging (VST) and Shared Uplink Sets together. Each of the steps is described in one of the sections below.
Configure VC for VLAN Mapping
Before you even create the Shared Uplink Set, you must first configure the Virtual Connect module to use VLAN mapping instead of VLAN tunneling.
To access the option for configuring VLAN mapping, you can use the menu bar across the top of the right side of the HP Virtual Connect Manager. Simply click Configure > Ethernet Settings, then click on the Advanced Settings tab. There you will see the option to either tunnel VLAN tags or map VLAN tags. Choose to map VLAN tags. Optionally, you can click the check box to force server connections to use the same VLAN mappings; this will eliminate a step later but limits the overall flexibility of the solution. It’s up to you if you want to use this option.
Create the Shared Uplink Set
Now you’re ready to create the Shared Uplink Set. Using the menu bar across the top of the right side of the HP Virtual Connect Manager interface, simply choose Define > Shared Uplink Set. Specify a name for the uplink set, then choose the external uplink ports. While you here, you can also go ahead and create the associated networks you’ll need later.
Create the Associated Networks
Either while you’re creating the Shared Uplink Set or after the Shared Uplink Set has been created, you can add the Associated Networks. While you are editing the Shared Uplink Set, simply click the Add Network button under the Associated Networks area and define one or more VLANs. The following parameters are available when you define an Associated Network:
- Network Name and VLAN ID: These are pretty self-explanatory. The VLAN ID here needs to match the external VLAN ID on the rest of the network.
- Native: If this VLAN is marked as the native VLAN on the rest of your network, check this box. This network would then receive all of the untagged traffic on the uplink set.
- Smart Link: If you would like this network to be marked as down if the uplinks go down, check this box.
- Private network: With this box checked, the network will act like a private VLAN–nodes on this network cannot communicate with each other.
- Advanced: This area allows you to set a custom speed for either the preferred speed or the maximum speed.
After you’ve defined the Associated Networks, then you’re ready to create the Server Profile—and that’s where the real magic in making VST is found.
Assign Networks in the Server Profile
Once again, you’ll use the menu bar across the right side of the HP Virtual Connect Manager to create a Server Profile. Simply select Define > Server Profile. In the Server Profile, you’ll add a network for each NIC present in the server blade (for a blade with Flex-10 NICs and a Flex-10 module, you will have eight NICs). When prompted for what network to associate to that NIC, choose Multiple Networks. Then click the small Edit button just to the right of the Network Name drop-down to show the Server VLAN to vNet Mappings screen.
The fact that you selected “Multiple Networks” when you added the connection to the Server Profile means that Virtual Connect will pass the VLAN tags up to the blade. The fact that you configured the Virtual Connect module to use VLAN mapping now means that you can create an association between the VLAN tags that a server uses and an corresponding Associated Network.
If you want to keep the server’s VLAN tags and the Associated Networks’ VLAN IDs matched up, just follow these steps:
- Check the box labeled Force Same VLAN Mappings As Shared Uplink Sets.
- Choose the Shared Uplink Set from the drop-down list.
- Place a check mark next to each Associated Network/vNet/VLAN. This tell the Virtual Connect module to include that VLAN.
- Place a check mark under Untagged for whichever Associated Network is should be handled as the untagged (native) VLAN.
If, on the other hand, you want to specify different server VLAN tags than Associated Network/vNet VLAN IDs, leave the check box for Force Same VLAN Mappings As Shared Uplink Sets unchecked, and specify the server VLAN ID that should be used for each Associated Network/vNet. For example, if the server was using VLAN ID 10 to refer to the Production network, but the Production network was using VLAN 1000 on the Associated Network and on the rest of the network, then choose the Associated Network that represents the Production network and specify a server VLAN ID of 10. This allows you to create a mapping between the VLAN tags the server uses and the VLAN tags the rest of the network uses.
After you’ve defined the Server Profile and created the vNet mappings, attach the Server Profile to a blade running VMware ESX/ESXi and you’re good to go! Within VMware ESX/ESXi, you would configure the vSwitches, distributed vSwitches, and port groups as you would normally.
How I Tested
My testing was performed on a HP Virtual Connect Flex-10 module running firmware revision 2.10. The Flex-10 module was uplinked via a single 10Gbps connection to an HP ProCurve switch. The blades were running VMware ESX 4.0.0.
Tags: Hardware, HP, Networking, Virtualization, VLAN, VMware
-
Keep in mind that you are currently limited to a maximum of 32 networks from within each Shared Uplink set. This poses a problem for a couple of my customers, and I much prefer to use VLAN tunneling for ESX environments on blades. Tunneling also prevents the administrative nightmare of having to go in and add/edit VLAN IDs inside Virtual Connect each time you add another VLAN to your network.
Sometimes, though, tunneling is not an option when a customer has a mix of ESX and Windows operating systems, the latter of which doesn’t support native VLAN tagging. In these scenarios, I lean towards the HP NIC Teaming utility.
my $.02,
Brian
-
Scott,
Important to note is that according to HP documentation, SmartLink is not currently supported with Flex10 NICs. In order to have fully redundant ESX vSwitches, HP recommends using Beacon Probing rather than link-state failure on the virtual switches. Being VMware guys, we know this means you need at least 3 network links per vSwitch. VMware’s KB article about this issue is article number 1007982. -
- The maximum networks for a shared uplink set is 64 not 32 (version 1.34 or 2.10 needed). There is still a restriction of 28 server VLAN-to-network mappings per server link.
- Another nifty feature when unchecking “force server connections to use the same VLAN mappings” is that you can feed a blade VLAN’s from multiple shared uplink sets.
-
Scott,
I’m trying to write a design for a vSphere implementation using BL685c G6 servers which have 4 Flex-NICS onboard, so I’ll see 16 NICS.
What I don’t know, because I don’t have the kit yet, is how these are mapped out.
If my blade enclosure contains 2 virtual-connect flex-10 modules, I’d like to have vmnics paired and going through seperate paths for redundancy.
I’m not planning to use all 16 NICS, only 8. Which also complicates the situation as I’m unable to say which 8 these will be and how to divide them up between my vSwitches.
Do you have any info on which LOM will hold which vmnics?
-
Very interesting document. Just the info i needed. As a Cisco guy, i was hesitating to put a vc domain in tunneling mode. I just want mapping mode AND the ability to make a trunk or standard access port. “Multiple VLANs” gives me just that. And from what i read here, i can even assign “Multiple VLANs” to a FlexNIC, and not the physical NIC. Nice. Regarding Smartlink, if you use the interconnect link and active/standby uplink sets, you don’t need smartlink feature….
I do have another question: if i have an uplink set containing interfaces across two VC modules, the ones on VC1 will be active, the ones on VC two will be standby. In HP, they are both assigned the same LAG ID. However, in my cisco uplink switch, uplinks on vc1 are on portchannel, my uplinks on vc2 are a different portchannel, so on cisco side, my lacp_ids are different. could this pose a problem ? -
Scott, how are you / others recommending you configure networking for Flex-10.
We are wanting to use a single 10GbE uplink from each Flex-10 module and I think I like the idea of keeping the network uplinks from module 1 and 2 separate and get the ESX hosts to handle the failover.
We are using NetApp and NFS for storage.
Would you use the Flex-10 capability of presenting multiple Nics to the hosts or just run everything over the two Nics, maybe making LAN/SC Primary on Nic1 and NAS/VMotion primary on Nic2 but using port groups so they could fail over to each other.
Should I use multiple VLANs to logically separate LAN/NAS/VMotion or again run everything over the same links.
Would you use shared uplinks or normal uplinks?
If you did have to carve out the Nics, what would you set and how would you lay it out.Would be great to see what people are doing for a reference design for ESX 4 on HP Blades.
Another problem we have come up with us which blade to use. Initially we liked the BL460C as it had hot swappable drives but in order to get a reasonable amount of memory you had to use 8Gb modules which brought up the cost significantly. We then veered towards the BL490c as it has more memory slots so you could use 4Gb modules which are cheaper. Even though you can use Solid state disks you cannot mirror them which creates a single point of failure so you may as well use the SD Card and ESXi but that is again a single point of failure and we’re not sure we’re comfortable with running ESX off one drive (opinions welcome).
So we’re back to the 460C with HSS drives and using only 48Gb Memory and having multiple blades as it’s cheaper running 2 x 460c with 4Gb Dimms than a single fully populated with 8Gb Dimms.So any opinions welcome.
Would love to hear how you are laying out your Flex-10, what you present to the hosts and whether you pass the VLANs directly to the host or create a virtual switch and how you split out traffic.
-
We are knee deep in a design/setup. A couple things I am struggling with are the Server profiles. I don’t understand why you have to name each server a different profile name. Somewhat anoying.
The other issue is we have 2 enclosures with 2 flex in each chassis, daisy chaining the modules between enclosures. Strange is if the 2 master modules are taken out or offlined (in chassis 1), you are orphaned from the other modules (chassis 2). What anoying is you would THINK you could get to the surviving modules to manage, but NO. yes, you still have connectivity through the surviving modules.
I am not sure if this is by design, or we are doing something wrong. We are new to the modules, so any help is appreciated.
oh, We have also gone with the BL460 G6’s with 48GB of ram. Seems to be the best bang for our buck, and will plan on putting the 8GB modules in when they come down. We run mostly windows boxes, so we run out of memory before procs etc. Also running Netapp 6040’s with PAM and dedupe. We are seeing 38 to 60 % savings on the space…
PEACE
-
We have set up our flex 10’s with an LACP port channel to each flex 10 module. The shared uplink sets are named like sus_VDC-Set1-A and
sus_VDC-Set1-A-B. Suppose these carry vlan ids 100,200, 300 and 400.
I define networks on each of the uplink sets as nw_100-A and nw_100-B for the respective uplink sets. Then when setting up NICs you choose an uplink set for the NIC and only select networks defined on that uplink set. This prevents loops as the left NIC’s go out sus A and the right NIC’s go out sus B. Then we make a failover ot TeLB bond for the NIC’s. Done! Dual redundant LACP paths back to different switches with the host responsible for failover.One other thing to think about is that if you alternate the sever profiles so that say everyblade in an odd slot uses sus A for the left NIC and every even slot uses sus B for the left NIC then you can get the incoming traffic to the blades coming in both sus’s rather than just the left without mucking around too much with preferences in the teaming config. Setting the first NIC always as the preferred master will then cause things to failover and fail back at the host level.
-
We’re setting things up similar to Greg, however I’ve run into an ODD issue where the ARP requests from a Guest VM to the Cisco Gateway is never getting back to the VM. I can use ICMP pings of other servers on the same VLAN (across switches mind you) and it works fine. Have a case open with Vmware on it, but haven’t heard back
-
Hello Chuck Hooper, how you resolve the issue with ARP?, i have the same problem.
Regards -
Hello Chuck, and Diego. We are having the same problem have you been able to resolve the issue. We have been working with HP, Cisco, and VmWare and no one seems to be able to find out what is happening.
Site Sponsors
13 comments
Comments feed for this article
Trackback link: http://blog.scottlowe.org/2009/07/06/using-vmware-esx-virtual-switch-tagging-with-hp-virtual-connect/trackback/