In an earlier article about VMware ESX, NIC teaming, and VLAN trunking, I described what the configuration should look like if one were using these features with Cisco switch hardware. It’s been a quite popular post, one I will probably need to update soon.
In this article, I’d like to discuss how to do the same thing, but using HP ProCurve switch hardware. The article is broken into three sections: using VLANs, using link aggregation (NIC teaming), and using both together.
Using VLAN Trunking
To my Cisco-oriented mind, VLANs with ProCurve switches are handled quite differently. Port-based VLANs, in which individual ports are assigned to one or more VLANs, allow a switch port to participate in that VLAN as either an untagged fashion or in a tagged fashion.
The difference here is really simpler than it may seem: the untagged VLAN can be considered the “native VLAN” from the Cisco world, meaning that the VLAN tags are not added to packets traversing that port. Putting a port in a VLAN in untagged mode is essentially equivalent to making that port an access port in the Cisco IOS world. Only one VLAN can be marked as untagged, which makes sense if you think about it.
Any port groups that should receive traffic from the untagged VLAN need to have VLAN ID 0 (no VLAN ID, in other words) assigned.
A tagged VLAN, on the other hand, adds the 802.1q VLAN tags to traffic moving through the port, like a VLAN trunk. If a user wants to use VST (virtual switch tagging) to host multiple VLANs on a single VMware ESX host, then the ProCurve ports need to have those VLANs marked as tagged. This will ensure that the VLAN tags are added to the packets and that VMware ESX can direct the traffic to the correct port group based on those VLAN tags.
In summary:
- Assign VLAN ID 0 to all port groups that need to receive traffic from the untagged VLAN (remember that a port can only be marked as untagged for a single VLAN). This correlates to the discussion about VMware ESX and the native VLAN, in which I reminded users that port groups intended to receive traffic for the native VLAN should not have a VLAN ID specified.
- Be sure that ports are marked as tagged for all other VLANs that VMware ESX should see. This will enable the use of VST and multiple port groups, each configured with an appropriate VLAN ID. (By the way, if users are unclear on VST vs. EST vs. VGT, see this article.)
- VLANs that VMware ESX should not see at all should be marked as “No” in the VLAN configuration of the ProCurve switch for those ports.
Using Link Aggregation
There’s not a whole lot to this part. In the ProCurve configuration, users will mark the ports that should participate in link aggregation as part of a trunk (say, Trk1) and then set the trunk type. Here’s the only real gotcha: the trunk must be configured as type “Trunk” and not type “LACP”.
In this context, LACP refers to dynamic LACP, which allows the switch and the server to dynamically negotiate the number of links in the bundle. VMware ESX doesn’t support dynamic LACP, only static LACP. To do static LACP, users will need to set the trunk type to Trunk.
Then, as has been discussed elsewhere in great depth, configure the VMware ESX vSwitch’s load balancing policy to “Route based on ip hash”. Once that’s done, everything should work as expected. This blog entry gives the CLI command to set the vSwitch load balancing policy, which would be necessary if configuring vSwitch0. For all other vSwitches, the changes can be made via VirtualCenter.
That’s really all there is to making link aggregation work between an HP ProCurve switch and VMware ESX.
Using VLANs and Link Aggregation Together
This section exists only to point out that when a trunk is created, the VLAN configuration for the members of that trunk disappears, and the trunk must be configured directly for VLAN support. In fact, users will note that the member ports don’t even appear in the list of ports to be configured for VLANs; only the trunks themselves appear.
Key point to remember: apply your VLAN configurations after your trunking configuration, or else you’ll just have to do it all over again.
With this information, users should now be pretty well prepared to configure HP ProCurve switches in a VMware ESX environment. Feel free to post any questions, clarifications, or corrections in the comments below, and thanks for reading!
Tags: ESX, Hardware, HP, Networking, Virtualization, VMware
-
Hi Scott,
Do you enable flowcontrol on the ProCurve switches. I have had some issues with IP storage, ESX, and flowcontrol on HP Procurves. My research came up with references to some procurves using a poor/non-standard implementation of flow control. Do you have any experience with this?
-
Does this work with ESXi as well ?
-
Francois,
I haven’t tested it with ESXi, but I don’t really see any reason why it wouldn’t work on any vSwitch other than vSwitch0. To change vSwitch0, you generally need command-line access, which of course isn’t possible with ESXi. In addition, you can’t use the Remote CLI because of network connectivity.
Otherwise, it should work fine.
Good luck!
-
Fransois
I currently have it running with ESXi and it works just beautifully. you can change the Vlan of vSwitch0 using the configurations menu on the ESXi.
Happy hunting.
-
Can you do link aggreation across multiple switches? In my scenario I am looking at a HP server wth 8 nics, 4 application and 4 storage. Using two HP ProCurve 3500YL’s, can I create one team for the application ports and spread them out over two switches? I know you can with Cisco when they are stacked, but I don’t know about doing this with HP???
-
Mark,
Though I haven’t tried it yet i think you can “Mesh” the two switches. once that is done I believe you can perform functions across switches.
-
Mark
You need to buy a Premium License for both 3500yl’s then you can use VRRP.
-
We have a Dell R900 with 4 onboard Broadcom Nics. We need to know hwo to disable ‘flow control’ on those nics. I am not a linux/unix guy, so please provide command that would allow this. Thank you in advance.
-
I just got ESXi set up, bought a Procurve 1800-8G as a result of this post, but the setup directions are still unclear to me.
I have an ESXi setup with 2 Intel Pro 1000 GT NICs. They’re both set up as VSwitch0, and they appear to be working well.
I’m still not entirely clear on what I must do on the HP Procurve 1800-8G side in order to get it set up correctly. I’ve tried turning on Trunk1 and associating it with ports 7&8 (where the 2 NICs for the ESXi is plugged into) and I then lose connectivity with the rest of my network.
The PC1800-8G setup is back to straight defaults except having jumbo frames turned on. What must be done first? Is it possible to simply post a step-by-step guide?
-
@John Smer
Just a guess, but check your VLAN configuration on “TrkX” after it is configured.
-
This is a great post, but there is a major GOTCHA which I couldn’t find anywhere else. You MUST set the trunk on EVERY VLAN on which it will receive traffic! So if you want the trunk to receive traffic for VLAN 5,6, & 7 then first setup the trunk, then add it to each VLAN with the ports tagged.
The command would be:
configure
vlan 5 tagged Trk1
vlan 6 tagged Trk1
vlan 7 tagged Trk1 (Use your own Vlan-id’s and trunk names of course)This article is also extremely helpful
http://docs.hp.com/en/J4240-90039/apds01.htmlI hope I just saved someone hours of work!
-
Trackback from Virtual RJ on Friday, January 2, 2009 at 7:10 pm
-
Well I finally took the time today to really learn about vlans and trunking cause man for some reason vlans and trunking confuse me. But thanks to this post its a little more clear. Thanks guys
-
I have comissioned ESX implimentations with Cisco switching for a number of years and have become acustomed to that way of working, however the network which I now manage is almost entirely ProCurve based.
In configuring pNIC teaming between ESX hosts and ProCurve 3500/5400/8200 family devices, I encountered an odd and frustrating issue with link aggregation (or ‘trunking’ as it is refered to in the ProCurve world).
The issue was that I would configure a 2 port (static - no negotiation protocol) trunk on a ProCurve 5400, i.e.
# trunk ethernet B20,D20 Trk25 trunk
I would then configure the appropriate vlans (all tagged) atop of the trunk interface, Trk25 in this example; with vlan 4001 being the service console/management network. The ESX server is configured to expect the service console/management traffic TAGGED (non-default).
(config)# vlan 4001
(config-vlan)# tagged Trk25
(config)# vlan 4002
(config-vlan)# tagged Trk25These two ports would be connected to an ESX host; Immediately I would loose all connectivity to the host… however if I disabled one of the two links (leaving the trunk config in place on the switch) connectivity returned to normal. I double checked the load balancing configuration on the vSwitch(0) of the ESX host - it was correctly set to ‘Route based on ip hash’ (non-default), but no dice…
The issue lay in that there were overriding load balancing configuration settings within the configuration of the ‘Management Network’ of the ESX host; these settings were the ‘defaults’ (i.e. route based upon virtual port ID), however they were configured (tick boxes) to override the configuration of the associated vSwitch! - the Management Network appeared to be set to override a fair number of vSwitch configuration options by default, and was causing all of my issues! Disabling all of these override options (which is appropriate in our configuration), so that the vSwitch configuration options are the only ones considered resolved the ‘no connectivity’ issues - obviously (well, now anyway) the Management Network was attempting to load balance in a means that the ProCurve switch could not handle - i.e. originating virtual port ID.
To summarise:
Switch configuration:
# trunk ethernet B20,D20 Trk25 trunk
(config)# vlan 4001 < Management Network vlan
(config-vlan)# tagged Trk25
(config)# vlan 4002 < Virtual servers VLAN #1 of x
(config-vlan)# tagged Trk25etc. etc.
ESX host configuration:
vSwitch0 configuration (’NIC Teaming’ tab):
Load Balancing: Route based on IP hash (non-default setting)
Network Failover Detection: Link Status Only
Notify Switches: Yes
Failback: Yes
Active Adapters: vmnic0, vmnic1
Standby Adapters: None
Unused Adapters: None.Management Network configuration:
VLAN ID: 4001
IP Address: x.x.x.x/yyLoad Balancing: UNDEFINED - inherit from associated vSwitch
Network Failover Detection: UNDEFINED - inherit from associated vSwitch
Notify Switches: UNDEFINED - inherit from associated vSwitch
Failback: UNDEFINED - inherit from associated vSwitch
Override vSwitch Failover Order: UNDEFINED - inherit from associated vSwitchThe same settings would also need to be made for any ‘VM Networks’ - the critical setting being ‘Load Balancing = Route based on ip hash’ or ensure the networks associated with the vSwitch are set to inherit the properties of the vSwitch.
A default setting which caused me a good couple of hours head scratching. Hope this info will be of use to others.
Tom
-
Finally I have a working virtual infrastructure lab with some old servers and a few hp 1800 (web managed) switches. But before I start learning other topics I have some questions on how vlans are used in a virtual scenario.
I’m still a bit confused on where you actually create the vlan.
From the post it seems that you do it at layer-2 with the port based tagging method, but then, you would need a port for each ingress connection, the switch would tag it and forward it to each corresponding 802.1Q trunk / tagged link.
Is this correct or am I missing something? Isn’t it like in the real world where you create the vlan at layer-3 and then use the switching infrastructure for forwarding? I’m asking this because you mention that the switch is tagging the packets that will reach the vSwitch, while to me it seems it just forwards packets based on the vlan id it finds in the tag.Since we are talking about port based vlans, the only case where my switch can modify a packet traveling the trunk is based on that port (PVID). This is the only way I can do it with my (cheap) equipment, but it also make me wonder if there is a better way of doing it, maybe a layer-3 switch?
Hope that somebody could clear this to me.
-
I have a ProCurve 5412zl and tryed everything above and cannot get the other vlans to work.
Trk1 - Trunk
vlan 0 (Default) - untagged
vlan 106 - tagged
vlan 110 - tagged
vlan 111 - tagged
vSwitches only see vlan 0What’s wrong?
-
Murray, Have you created the VLAN on the virtual networks you want to see the tagged VLANS on? I think that may be necessary or the virtual networks will only see the untagged VLAN otherwise.
I could be wrong however.
Site Sponsors
18 comments
Comments feed for this article
Trackback link: http://blog.scottlowe.org/2008/09/05/vmware-esx-nic-teaming-and-vlan-trunking-with-hp-procurve/trackback/