I wanted to post a quick follow-up on the previous two articles that I published regarding using VLANs, Virtual Switch Tagging, HP Virtual Connect, and Flex-10:
Using VMware ESX Virtual Switch Tagging with HP Virtual Connect
Using Multiple VLANs with HP Virtual Connect Flex-10
One thing that I wanted to really clarify was that you can present multiple VLANs to FlexNICs on the same LOM, but you can’t present the same set of VLANs to FlexNICs on the same LOM. I’m not sure that I made that clear enough in my other post. So, if you have VLANs 100, 101, 102, 103, and 200, you can present any number or combination of those to all the FlexNICs on a single LOM—but it must be in a non-overlapping configuration, so that the same VLAN isn’t presented to multiple FlexNICs on the same LOM. I plan on posting some sample configurations, with graphics, that should clarify things even more.
The other thing I wanted to clarify was why I posted an article about presenting multiple VLANs to FlexNICs on the same LOM. Usually, this topic comes up in the form of a question (and I’ve had several readers e-mail me about this) like this: “You state that you can’t present the same set of VLANs to multiple FlexNICs on the same LOM. Why in the world would you want to do that?”
That’s a good question! In a Flex-10 environment, you normally wouldn’t want to do that, and not just for the reason that it doesn’t work (although having a configuration that works is usually quite beneficial). Consider the value proposition behind Flex-10: it provides four logical NICs (the FlexNICs), each of whose bandwidth can be adjusted as needed, up to 10Gbps, based on the traffic requirements. Now consider the reason why administrators normally use multiple NICs: more bandwidth. Considering that you can allocate bandwidth easily with a FlexNIC, there’s no need to use multiple FlexNICs on the same LOM to handle the same VLANs. A single FlexNIC can handle multiple VLANs just fine because you can allocate more bandwidth to that FlexNIC easily. And since presenting multiple VLANs to FlexNICs on different LOMs works just fine, you can use one FlexNIC from each LOM to gain redundancy. All the reasons for wanting to use multiple NICs are addressed by using only two FlexNICs, one from each LOM, and presenting as many VLANs as you like to those two FlexNICs.
However, having said all that, it is the default configuration in many VMware environments to present VLAN trunks to all ports on all ESX/ESXi hosts (i.e., to present the same set of VLANs to all ports). In a Flex-10 environment, you’ll have to break out of that line of thinking. Hence, why I posted the information that I posted, so that VMware administrators would realize they can’t follow the same configuration guidelines in the Flex-10 environment as they would follow in a more “traditional” networking environment. Just as virtualization with VMware requires server admins to approach things differently, the functionality offered by Virtual Connect also requires server admins to think a bit differently about how network connectivity is presented to VMware ESX/ESXi hosts.
Have more questions, or need additional clarification? Speak up in the comments. Thanks for reading!
Tags: Hardware, HP, Networking, Virtualization
-
Scott,
Based on what you have described here it sounds as if there is no tagging mechanism that identifies which FlexNIC the traffic is either destined to, or sourced from?It sounds like Flex-10 / FlexNIC uses the VLAN tag as the means to provision and identify what FlexNIC a frame belongs to.
If an additional tag was used, such as a VNTag, Flex-10 would not have this limitation.
Cheers,
Brad -
Brad, as I understand it, the FlexNICs are PCIe functions–not SR-IOV, but similar. Therefore, their functionality would reside well below VLAN tags. I suspect that the issue with VLAN tags is more due to the fact that HP didn’t see the need to have multiple FlexNICs on the same LOM support the same VLANs–why use multiple FlexNICs on the same LOM when you can just assign more bandwidth to one FlexNIC? I think this was an operational decision by HP, not a technical limitation. Of course, I’m just theorizing…someone from HP is welcome to chime in!
-
Scott,
Perhaps I could be more clear. When a frame is received by the LOM, the only way it knows to which FlexNIC that frame belongs is via the VLAN tag. Hence the unique VLAN per FlexNIC per LOM limitation.How the adapter presents itself as multiple adapters to the system (PCIe or SRI-IOV) has nothing to do with how the adapter is viewed as multiple adapters to the upstream network device(s) (Flex-10).
Brad
-
” … why use multiple FlexNICs on the same LOM when you can just assign more bandwidth to one FlexNIC?”
One obvious use case that comes to mind: Multiple FlexNICs on the same LOM on the same VLAN would allow for Virtual Machines to use the FlexNICs as their NIC … example VMDirectPath.
It sounds as if with FlexNIC / Flex-10 you will not be able to use a pass-through vSwitch or VMDirectPath, you will always need a full blown software switch running in the hypervisor.
Brad
-
Brad,
Each FlexNIC has its own unique MAC address, so I still don’t see how the VLAN issue could be related. To the operating system, it appears as a unique physical network adapter; to the network, it appears as a unique physical network adapter. Again, I think that this was a design decision, not a technical limitation.
As for your second comment regarding VMDirectPath, keep in mind that hypervisor bypass (i.e., VMDirectPath) is not necessarily the “be all end all” configuration. There is still a tremendous amount of value to be had in running a software switch in the hypervisor. Yes, there are configurations and workloads that will benefit from hypervisor bypass, but that doesn’t mean that EVERY configuration and EVERY workload needs–or even wants–hypervisor bypass. I know that you (and Cisco) think that this is the best configuration and the perfect long-term solution, but many people still believe (and I agree) that value remains in having a software switch running in the hypervisor.
-
Scott,
The destination MAC address of a frame received by the Flex LOM will not always be that of a FlexNIC, especially in a VMware environment where the destination MAC could belong to a VM. So, either the Flex LOM is acting like a switch and learning which MAC’s belong to which FlexNIC (unlikely) – Or – more likely, the Flex LOM is looking at the VLAN tag to associate received traffic to a FlexNIC, since no other tag is employed such as a VNTag.
As for your statement about the value of a hypervisor switch, believe or not, you and I (and Cisco) are in complete agreement – hence the development of the Nexus 1000V.
Speaking for myself, I also think there is value in having an infrastructure that is adaptable to either a hypervisor switch or hypervisor bypass configuration.
Cheers,
Brad -
With regard to the behavior of a FlexNIC in a VMware environment, how is that any different from an “ordinary” NIC? An ordinary NIC must accept traffic for MAC addresses that do not belong to that NIC; the same goes for a FlexNIC. I guess I’m dense, but I don’t see a difference here.
-
Scott,
There is a key difference here between an “ordinary” NIC and the FlexNIC … The “ordinary” NIC has a it’s own dedicated cable. The FlexNIC, on the other hand, is sharing the same cable with other FlexNICs. Therefore, when a packet arrives on the shared cable, the physical adapter must first figure out which FlexNIC the packet belongs to.
This is not an easy problem to solve. To do it the right way requires a new tag in the packet, or it requires an adapter that acts more like a network switch through learning and organizing tables of MAC addresses.
What you have discovered here is that HP has done neither of the two. Rather, it appears HP took the easy way out and decided to use the existing VLAN tag as the means to decide which FlexNIC owns the packet arriving on the shared cable. This is not true network interface virtualization as it requires that a VLAN be unique to only one virtual adapter (FlexNIC).
Ture network interface virtualization (NIV) must be addressed from both the system side and the network side. Ignore one of the two and you will be faced with quirky limitations and challenges such as you have discovered here.
Cheers,
Brad -
Scott,
We’re using BL685c G6’s with 4 LOMS giving us 16 FlexNICs.
Each chassis will have a pair of VC-Flex-10 modules each containing 4 1Gbp/s uplinks (we have no 10GbE available).
Knowing the limitation of 64 VLANS when using VLAN tagging, we need to use VLAN tunnelling.
I understand that you can’t map the same VLAN onto two different FlexNICS from the same LOM. Which basically means if you’re using the multiple networks option, you can only ever present multiple VLANs to 4 LOMS (4 NICS basically).
So what value am I actually getting from Flex-10?
In terms of bandwidth allocation, I’ll have a maximum of 4GB coming out of each VC which needs to be shared between the service console, vmotion, vm networks and backups.
If I decided not to use this and go for traditional VC interconnects, would I be delivering the same solution for less cost?
Stuart
-
“Flex-10 technology technical brief, 1st edition” (http://tinyurl.com/mf5zrz) provides a good overview of how Flex-10 works I think. The “issue” with presenting the same network to multiple FlexNIC’s are also documented here.
-
Stuart,
Let me take a stab at your situation. There are a couple of things to consider here. The first is whether you need the extra NIC ports. If you need more than 4 NICs, then Flex10 FlexNIC capability is a decision point. Flex10 will give you 16 FlexNICs rather than 4 standard NICs. Although you may not like the architectural limitation of not being able to assign the same VLANs to multiple FlexNICs on the same LOM (LAN on Motherboard), you still get some cool benefits…
The 10Gb FlexNIC bandwidth is still usable for critical functions like VMotion even if you have 1Gb uplinks. If you are using multiple enclosures, Virtual Connect modules can be “stacked” across chassis so that you have a 10Gb backbone across up to four chassis. The stacking links go between chassis and do not require any ports at your switching core. In the case of VMotion, you can create a VMotion vNet in Virtual Connect, and assign the various FlexNICs more bandwidth on that vNet connection. That will give you faster VMotion capabilities within or even across enclosures.The traditional Virtual Connect modules will only provide 1Gb downlinks to the servers, so your 10Gb NICs will be limited at 1Gb – including VMotion, and you do not want that. The only way to extract the 10Gb speed out of the ports is to use either Flex10 or the HP 10Gb Ethernet BL-c Switch.
Bottom line the Flex10 will give you faster, cross chassis VMotion capabilities at the very least. Yes, if you want to have more VLANs than are currently supported, you will need to use VLAN tunneling, but you can still use the 10Gb VMotion strategy.
-
Stuart,
One more thing – I would suggest posting technical questions like this on the HP Blade Interconnect site in the Virtual Connect section. that is the ideal forum to get feedback from Virtual Connect savvy people.
http://h18006.www1.hp.com/products/blades/components/bladeconnect.html
-
Just a general comment on the content here –
Since Flex-10 is 100% hardware transparent (no drivers or O/S awareness required), it will work not only with VMware, but Hyper-V, XenServer, and any other virtualization platform which becomes available on the x86/x64 platform. The only thing required is the standard 10Gb NIC driver.
As Scott mentioned, FlexNICs appear as standard PCIe functions, allowing the OS to see them as regular NICs. This allows for a great deal of Flexibility (no pun intended) when it comes to supporting any operating system or virtualization platform.
We can’t forget that although VMware is the market share leader, many companies are running or evaluating multiple virtualization strategies. Virtual Connect Flex10 will work transparently across any of them.
-
Scott,
Do any one have a setup of vSphere/NK100V on Flex 10 with an uplink to Nexus 5000 ?
Thanks
Moges
-
As far as i can tell, from the discussion going on here, there is some confusion about how VLANs operate in a normal network. And, particularly, how they are managed by Spanning Tree.
VLANs operate on layer two. They have no TTL like packets (layer 3) and rely on a single path up through the infrastructure (via trunk connections between switches), until they get to the core of the network (or the point at which VLANs are terminated and the path beyond is routed).
To me, a clear reason why physical, or virtual Flex, connections cannot carry the same VLAN up two separate uplinks is due to the essential operation of VLANS and Spanning Tree. You CAN have two uplinks that can carry the same VLANs, just not at the same time. In other words, the secondary link (whether physical or virtual) must be in standby.
If you have one VLAN running on two separate uplinks at the same time, you are going to get a Spanning Tree loop. I don’t see any reason why HP or anyone else is going to be able to solve a technology problem that has existed since VLANs existed. No other network company has been able to provide an active/active solution for layer two infrastructures.
Does this make sense in the context of this discussion (I’ve kinda just jumped in and am only now looking at VC in a virtualized environment.
-
One way that the industry was able to make better use of dual uplinks (in a traditional physical network most network devices are tiered with dual uplinks) is to enable each uplink to carry a different set of VLANs. This is enabled through the use of Per VLAN Spanning Tree (PVST+).
The next improvement to the technology was to develop MST (Multiple Spanning Tree). This allows easier management of groups of VLANs.
It is ALL about creating stable and efficient pathing. Virtualization is simply about abstraction and the underlying realities still apply. While we use VLANs, we have to understand how they are managed in the normal world. Do some research into VLANs, STP and Rapid-PVST+ and MST.
Then, when you come back to your virtual environment, you will be able to relate everything back to the technologies that were created in the physical universe and it will be much more straight forward.
-
I just re-read my statement: “VLANs operate on layer two. They have no TTL like packets (layer 3)…”.
I should clarify: Frames (which encapsulate IP Packets passed down from layer three) operate on layer two and are constrained “within VLANs”. Frames have no TTL, hence Frames do not timeout and therefore are not removed from the network (whereas packets are removed once their TTL is reached). If you have a pathing loop somewhere, the frames just keep going around forever. Enter Spanning Tree Protocol.
Actually, Frames are also passed between Routers. In this case, however, there is no VLAN information (tagging) and the IP Packets are encapsulated just to get them onto the link which connects to the next Router in the path (route).
So, I may have inadvertently given the impression that VLANs are the basis for layer two Ethernet network technologies. When, in fact, Frames (and switching/bridging devices) form the basis for layer two operation (on an Ethernet network). I hope this doesn’t further confuse things (the concepts are simple, the details are not).
I think Server guys are going to have to learn more and more about networking technologies, as their domain (traditionally just endpoints on the network) and now becoming part of the network. We are no longer just teaming a few NICs.
-
With all these limitations mentioned above, why can’t we implement Rate Limiting Function of dVS in vSphere to do the bw allocation / traffic shapoing to get dedicated bw for types of traffic& then use VLAN tagging, port grouping, and VST for taffic isolation, in combinitation with teaming two normal 10Gig NICs that could work with any other Standard Switch out there. What is the real advantage of locking into a proprietary interconnect solutions, when the same can be achieved otherwise with available funtionality of dVS, Stand NICS and Switches.
Can somebody enlighten me why use Flex-10 and VC, when standard solutions are moving towards SR-IOV and DCB for FCoE….
VC and Flex-10 has all kinds of other limitations that effect advanced functions that were built in to existing network infrastructure such as QoS, ACLs, VLANs, Multicasting, Security etc…you have to change the way these advanced funtions are used to maximum capability because limitations at the server edge. Do you really think this is a right interconnect connecting your enterprise Server Farm? And are willing to lock down to a proprietary solutions and join your IT to HP at the hip???
I still don’t get it? Can one you shed some light and so that I could over come my doubts about this VC-Flex10 solutions…
-
SRT,
I deleted the redundant comments on other posts; your questions/comments are most relevant here.
If you are already using HP blade servers, then Flex-10 is a good solution. You’ll need switches in the back of the blade chassis anyway, and the Flex-10 modules will uplink to any compatible 10Gbps Ethernet switch (albeit you might have some difficulties with SFPs, but that’s another story). The “proprietary-ness” of Flex-10 is no different than the “proprietary-ness” of Cisco’s I/O Modules in a Cisco UCS solution. It’s also no different than the “proprietary-ness” of the Cisco Nexus 4000 for the IBM blade chassis. HP chose to use a different method of presenting NICs than SR-IOV, and it has its advantages and disadvantages. Cisco chose to use SR-IOV, and it has its advantages and disadvantages. Every technology decision has its tradeoffs…even “standards-based” decisions.
-
Scott,
Thanks for responding quickly. I really appreciate it.I still did not get those two good reasons why Flex-10 is needed and is a good solution, even if you already have BladeSystem. The capability to divide 10G in to four FlexNICs is presented by HP as the solution needed for virtualization to meet traffic isolation and dedicated BW requirements. Instead of using multiple 1Gig NICs they suggest use a 10Gig Flex-10 NIC and slice it in to 4 FlexNICs. Then use these separated FlexNICs for meeting dedicated BW requirements. Instead I can use vDS functions such as Rate limiting to do get dedicated BW. So I don’t think Flex-10 has any value proposition here.
My other issue is, you can leverage this FlexNIC capability with only VC modules, since both have BroadCom chips that are compatible to enable this function. It doesn’t work with any other HP blade switch or to that matter any other top of the rack switch. So now tying me down to VC only option, running me into all the limitations of VC. So instead of value add here, I am actually downgrading my network capabilities if I end up with VC.
So when I ran in to your blog, I thought I did know something that you know about Flex-10 and VC, and that is why you are for it. Now I am a little disappointed? that you did not present those additional value propositions this technology is bringing to Virtualization, than those other technologies already did. A value proposition may drive my decision to adapt Flex-10 VC, but what is it??
So, I think it is better to have normal 10G NIC in the server and connect to standard L2 switch, enable vDS existing features such as Rate limiting, Port Groups, VLAN tagging….this way you get all the consolidation (of Servers, of NICs, of Switches, etc) using virtualization and get overall Data Center consolidation without locking down further proprietary solutions….This way I am ready to take advantage of future IO and Fabric Consolidation as standards ratify early next year or so…and not worry about interoperability issues in my Data Center..
I would be glad to hear more about the really Value Proposition of Flex10 VC…
Anyone?? Please share…
-
Although the discussion here is healthy, it’s more academic than useful in the real world. The Flex-10 is the absolute worst piece of network gear I’ve ever tried to work with. It has so many limitations that it’s almost impossible to use in all but the most simple of environments. Limitations like:
* Bandwidth assignments to FlexNICs are “hard” meaning if you assign 1Gbps to one FlexNIC and 4Gbps to another, and the 4Gbps isn’t using all of the 4Gbps, that “left over” bandwidth CANNOT be used anywhere else, meaning you could be dropping other production traffic without the 10Gbps interface being throttled. Poor design.
* Only up to 28 VLANs per FlexNIC if VLAN tagging!!
* Because of it’s proprietary loop prevention, a tunneled FlexNIC it cannot have a MAC address in it’s CAM table in more than one VLAN. This is a MUST for any environment using bridged server-load balancers (the most common method for deploying SLB)
* It can only have 128 VLANs on the whole module (tunneled FlexNICs do not count toward this 128), which if you’re bonding uplinks, each VLAN counts multiple times for each link (i.e. if you a 30 VLANs, and you have 2 x 10G uplinks bonded, that counts as 60 VLANS!)
* It supports NO sort of Layer 2 QoS (CoS) meaning you’d better not ever want to try to put any voice applications (i.e. IP PBX, Voicemail, etc) on a blade. Even worse if you’re trying to do a virtual environment using iSCSI or NFS for your storage protocols.
* If a Flex-10 loses it’s uplink it doesn’t have the ability to down the internal server links, meaning the server has NO way of knowing if it’s upstream Flex-10 is a black hole. (and it can be a black hole depending on how you configure multiple Flex-10s).HP has this protocol they came up with to solve this last issue, called SmartLink. SmartLink, upon a Flex-10 losing it’s uplink(s), sends a signal to the host telling to down it’s NIC. HOWEVER, for SmartLink to work, your OS on the blade MUST have a driver that supports SmartLink, which ESX, HyperV, nor XenServer have making SmartLink USELESS if you’re running a virtual environment!
I could carry on more, but I shant. Just the above should be enough for anyone considering deploying Flex-10 in a large data center to steer clear.
-
Gold Chain,
I am pretty much on board with your post but most of the conversation is more than academic. Having fought with this technology all of the posted concerns/issues come into to play somehow.
As for you last statements about SmartLink you are dead on! However VMWare does now have a Host based driver that allows this to work. I have a very large client that was having random percentages of their Traffic Black holed when we simulated a Flex10 Failure or a full connectivity failure to a Flex-10 module. Until the new host driver was applied to ESX the only solution to force traffic to the still connected Flex-10 unit was to Physically pull the isolated Flex-10 mod.
Still not a fan of Flex-10 but at least we are starting to see support for SmartLink to prevent host/guest isolation.
Scott,
Bravo man. You have generated some of the most intelligent and educational debate on this issue.
-
The only quirk I dislike about the Flex10 VC modules is the vlan limitation in a vnet. I’m hoping that a firmware update from HP will resolve that issue. We have a dz or so server based vlans here so i’m not worried about it.
All other quirks are no different than running bare-metal systems.
VmPassThru is a waste of resource to me. My manager places utilization demands upon our team. Resources aren’t purchased until we prove that bandwidth/cpu/memory are fully consumed.
I run a 6 node Oracle RAC cluster that is spanned across two c7000 chassis and four VC modules. The blades use two flexnics – one from each VC module. I have bond mode 1 and large mtu configured on a single bond interface which consists of the 2 flexnics. 3 vlan tagged interfaces are created on the bond interface.
When I require more bandwidth I’ll be purchasing additional sfp’s and those ports will be configured for etherchannel. Half my bandwidth is currently idle but in most environments that is the case. I tested etherchanneling my two VC’s and did some resiliency testing. All tests were successful. In fact you can do mode 4 bonding on the os level when VSS is implemented on upstream switches or if you have stack-wise switches installed. We don’t. Our uplinks are connected to two 6500′s.
This is the best Oracle cluster I’ve seen in my 6 years of dealing with RAC. I’m excited to get two more chassis so we can span the cluster across our wan.
Again, great post and conversation.
/Chris C -
Simple question from a NOOB. 2 c7000′s with redundant Flex 10′s stacked. 4 to 5 ESX BL460 G7′s in each all on the same VLAN. Can the VM’s in C7000 #1 send and recieve traffic to VM’s on C7000 #2 if the they are stacked.
My gut says yes because Flex is a layer 2 device. I am trying to save bandwidth on the uplinks and maximize the internal 10G network on the chassis.
-Rob
-
My question is more security based. I would like to completely know how the VC forwards packets. Lets say I have a mixed environment internal resources and dmz on the same VC.
All the trunks go to the VC and it is all separated out through the manager. What would the VC do if I send a packet to the wire with a vlan tagID of a internal network? Is it smart enough to know that this is coming from the DMZ and to not pass the traffic. I would think not the way you guys are talking about how dumb the VC’s are. -
Rob,
Yes, you can. It’s quite common to have vmotion network across chassis just use the stacked links and a non-routable subnet. So it will act as a straight switch. It’s a bit non-intuitive to configure.Anthony
-
Scott – Is there any known limitation to vlans on the VC
Flex 10 module when tunnelling? We are experiencing odd network
disconnects in our c7000 chassis with BL460c servers running ESX
vSphere 4.0 U2. We have some NFS datastores exported from NetApp
3170′s that are dropping and coming back. The same ones work just
fine from our DL380′s. I’d be glad to provide you with more detail
if you have a few moments to spare on this one. Thanks! -
Anthony
Flex-10 handles OS-tagged and untagged packets in mapping mode. Virtual Connect adds and translates the VLAN tags as the packet moves from the OS through the FlexNIC, the Flex-10 module, and then out to the external network. The FlexNIC adds its tag that identifies the FlexNIC that the packet came from.Virtual Connect combines that outer FlexNIC VLAN tag and the inner OS tag to map an Ethernet packet to a particular vNet.
Amit
-
Sorry my comments are for Dion.
-
silly question here about “its not possible to have traffic on same vlan on multiple uplinks at the sametime”
So what happens if i have two uplinks to esx with vlans 1 and 2 allowed on them
I have a vm in vlan 1 using uplink 1 and another vm in vlan 1 using uplink2. That means traffic from the same vlan is on two different uplinks
Usually esx uplinks have same vlans trunked on all of them as the uplinks are part of a team, and vms on the same vlan can use different uplinks but reading the posts above this should cause loops..
Site Sponsors
30 comments
Comments feed for this article
Trackback link: http://blog.scottlowe.org/2009/07/09/follow-up-about-multiple-vlans-virtual-connect-and-flex-10/trackback/