Comments on: Follow-Up About Multiple VLANs, Virtual Connect, and Flex-10

By: mike

mike — Sun, 18 Dec 2011 15:00:06 +0000

silly question here about “its not possible to have traffic on same vlan on multiple uplinks at the sametime”

So what happens if i have two uplinks to esx with vlans 1 and 2 allowed on them

I have a vm in vlan 1 using uplink 1 and another vm in vlan 1 using uplink2. That means traffic from the same vlan is on two different uplinks

Usually esx uplinks have same vlans trunked on all of them as the uplinks are part of a team, and vms on the same vlan can use different uplinks but reading the posts above this should cause loops..

By: Amit

Amit — Wed, 28 Sep 2011 12:21:45 +0000

Sorry my comments are for Dion.

By: Amit

Amit — Wed, 28 Sep 2011 12:20:52 +0000

Anthony

Flex-10 handles OS-tagged and untagged packets in mapping mode. Virtual Connect adds and translates the VLAN tags as the packet moves from the OS through the FlexNIC, the Flex-10 module, and then out to the external network. The FlexNIC adds its tag that identifies the FlexNIC that the packet came from.Virtual Connect combines that outer FlexNIC VLAN tag and the inner OS tag to map an Ethernet packet to a particular vNet.

Amit

By: Chris Bogart

Chris Bogart — Thu, 06 Jan 2011 18:55:23 +0000

Scott – Is there any known limitation to vlans on the VC
Flex 10 module when tunnelling? We are experiencing odd network
disconnects in our c7000 chassis with BL460c servers running ESX
vSphere 4.0 U2. We have some NFS datastores exported from NetApp
3170′s that are dropping and coming back. The same ones work just
fine from our DL380′s. I’d be glad to provide you with more detail
if you have a few moments to spare on this one. Thanks!

By: Anthony Skipper

Anthony Skipper — Tue, 02 Nov 2010 15:27:15 +0000

Rob,
Yes, you can. It’s quite common to have vmotion network across chassis just use the stacked links and a non-routable subnet. So it will act as a straight switch. It’s a bit non-intuitive to configure.

Anthony

By: Dion

Dion — Wed, 27 Oct 2010 17:18:06 +0000

My question is more security based. I would like to completely know how the VC forwards packets. Lets say I have a mixed environment internal resources and dmz on the same VC.
All the trunks go to the VC and it is all separated out through the manager. What would the VC do if I send a packet to the wire with a vlan tagID of a internal network? Is it smart enough to know that this is coming from the DMZ and to not pass the traffic. I would think not the way you guys are talking about how dumb the VC’s are.

By: Rob

Rob — Sat, 02 Oct 2010 00:15:47 +0000

Simple question from a NOOB. 2 c7000′s with redundant Flex 10′s stacked. 4 to 5 ESX BL460 G7′s in each all on the same VLAN. Can the VM’s in C7000 #1 send and recieve traffic to VM’s on C7000 #2 if the they are stacked.

My gut says yes because Flex is a layer 2 device. I am trying to save bandwidth on the uplinks and maximize the internal 10G network on the chassis.

-Rob

By: Chris C

Chris C — Wed, 15 Sep 2010 20:33:08 +0000

The only quirk I dislike about the Flex10 VC modules is the vlan limitation in a vnet. I’m hoping that a firmware update from HP will resolve that issue. We have a dz or so server based vlans here so i’m not worried about it.

All other quirks are no different than running bare-metal systems.

VmPassThru is a waste of resource to me. My manager places utilization demands upon our team. Resources aren’t purchased until we prove that bandwidth/cpu/memory are fully consumed.

I run a 6 node Oracle RAC cluster that is spanned across two c7000 chassis and four VC modules. The blades use two flexnics – one from each VC module. I have bond mode 1 and large mtu configured on a single bond interface which consists of the 2 flexnics. 3 vlan tagged interfaces are created on the bond interface.

When I require more bandwidth I’ll be purchasing additional sfp’s and those ports will be configured for etherchannel. Half my bandwidth is currently idle but in most environments that is the case. I tested etherchanneling my two VC’s and did some resiliency testing. All tests were successful. In fact you can do mode 4 bonding on the os level when VSS is implemented on upstream switches or if you have stack-wise switches installed. We don’t. Our uplinks are connected to two 6500′s.

This is the best Oracle cluster I’ve seen in my 6 years of dealing with RAC. I’m excited to get two more chassis so we can span the cluster across our wan.

Again, great post and conversation.
/Chris C

By: Josh OBrien

Josh OBrien — Thu, 26 Aug 2010 14:28:42 +0000

Gold Chain,

I am pretty much on board with your post but most of the conversation is more than academic. Having fought with this technology all of the posted concerns/issues come into to play somehow.

As for you last statements about SmartLink you are dead on! However VMWare does now have a Host based driver that allows this to work. I have a very large client that was having random percentages of their Traffic Black holed when we simulated a Flex10 Failure or a full connectivity failure to a Flex-10 module. Until the new host driver was applied to ESX the only solution to force traffic to the still connected Flex-10 unit was to Physically pull the isolated Flex-10 mod.

Still not a fan of Flex-10 but at least we are starting to see support for SmartLink to prevent host/guest isolation.

Scott,

Bravo man. You have generated some of the most intelligent and educational debate on this issue.

By: GoldChain

GoldChain — Tue, 23 Feb 2010 14:43:37 +0000

Although the discussion here is healthy, it’s more academic than useful in the real world. The Flex-10 is the absolute worst piece of network gear I’ve ever tried to work with. It has so many limitations that it’s almost impossible to use in all but the most simple of environments. Limitations like:

* Bandwidth assignments to FlexNICs are “hard” meaning if you assign 1Gbps to one FlexNIC and 4Gbps to another, and the 4Gbps isn’t using all of the 4Gbps, that “left over” bandwidth CANNOT be used anywhere else, meaning you could be dropping other production traffic without the 10Gbps interface being throttled. Poor design.
* Only up to 28 VLANs per FlexNIC if VLAN tagging!!
* Because of it’s proprietary loop prevention, a tunneled FlexNIC it cannot have a MAC address in it’s CAM table in more than one VLAN. This is a MUST for any environment using bridged server-load balancers (the most common method for deploying SLB)
* It can only have 128 VLANs on the whole module (tunneled FlexNICs do not count toward this 128), which if you’re bonding uplinks, each VLAN counts multiple times for each link (i.e. if you a 30 VLANs, and you have 2 x 10G uplinks bonded, that counts as 60 VLANS!)
* It supports NO sort of Layer 2 QoS (CoS) meaning you’d better not ever want to try to put any voice applications (i.e. IP PBX, Voicemail, etc) on a blade. Even worse if you’re trying to do a virtual environment using iSCSI or NFS for your storage protocols.
* If a Flex-10 loses it’s uplink it doesn’t have the ability to down the internal server links, meaning the server has NO way of knowing if it’s upstream Flex-10 is a black hole. (and it can be a black hole depending on how you configure multiple Flex-10s).

HP has this protocol they came up with to solve this last issue, called SmartLink. SmartLink, upon a Flex-10 losing it’s uplink(s), sends a signal to the host telling to down it’s NIC. HOWEVER, for SmartLink to work, your OS on the blade MUST have a driver that supports SmartLink, which ESX, HyperV, nor XenServer have making SmartLink USELESS if you’re running a virtual environment!

I could carry on more, but I shant. Just the above should be enough for anyone considering deploying Flex-10 in a large data center to steer clear.