Last year, I wrote a piece about multi-hop Fibre Channel over Ethernet (FCoE) and some of the various reasons why—at the time—multi-hop FCoE was not a practical reality. Some things have changed since I last discussed multi-hop FCoE, and today I’d like to take a quick look at the interaction between network interface virtualization (NIV) and FCoE and see how this plays into multi-hop FCoE.
If you haven’t already read my recent article on understanding NIV, go read it now. Likewise, if you haven’t read the multi-hop FCoE article, you should go read it now too. Believe me, the rest of this article will make a lot more sense if you do.
Done now? Good. Let’s get started.
From my article on multi-hop FCoE, I identified two key roadblocks to multi-hop FCoE support:
- First, there was a lack of widespread support for FCoE Initialization Protocol (FIP). FIP allows an FCoE initiator to be separated from a Fibre Channel forwarder (FCF) by one or more IEEE DCB-capable switches. This has since been addressed by updates to NX-OS (the code that runs on Cisco’s Nexus 5000 switches) and by Generation 2 converged network adapters (CNAs); both of these components now have FIP support included.
- Second, there was no defined standard for creating the FCoE equivalent of trunking E_ports (which I believe are referred to as VE_ports). VE_ports are necessary to link multiple FCFs together, much in the same way you would use an inter-switch link (ISL) in traditional Fibre Channel storage area networks (SANs). To my knowledge, this issue has not yet been addressed.
At first glance, NIV doesn’t seem to help with either of these roadblocks. When you take a deeper look, though, you’ll see that NIV can actually serve as a workaround for both problems. Here’s how.
In NIV, recall that you have both an interface virtualization (IV)-capable bridge (or switch) as well as one or more interface virtualizers (IVs). (Remember that IVs are also referred to as fabric extenders.) Network interface cards (NICs) connect to ports on the IVs, and the IVs uplink to the IV-capable bridge. Even though the IV-capable bridge and the IVs are physically separate devices, they appear as a single device. Even though there is a connection—typically an Ethernet connection—between the IVs and the IV-capable bridge, it appears and functions as a single device.
With this in mind, then, I’ll ask this question: what is a multi-hop topology? If a multi-hop topology is multiple physical devices connected over an Ethernet uplink, then multi-hop FCoE is possible today with an FCoE-enabled IV-capable bridge and one or more FCoE-enabled IVs. In fact, this is the topology that Cisco uses in its Unified Computing System (UCS): a pair of FCoE-enabled IV-capable bridges (the UCS 6100XP fabric interconnects) connected to one or more FCoE-enabled IVs (the I/O Modules in the back of the chassis).
Applying this line of thinking to our roadblocks above, we see that the use of NIV allows for greater port densities; greater port density is one of the primary reasons why users would want FCoE initiators separated from an FCF by an IEEE DCB-capable switch. In addition to leveraging FIP (and eventually leveraging the IEEE DCB standards once they are finalized), you can build the same sort of topology using an IV-capable bridge and one or more IVs.
Similarly, using NIV as a way of connecting multiple devices together eliminates the need to chain multiple FCFs together; this, in turn, eliminates the need for the FCoE equivalent of ISLs and the need to create VE_ports between the FCFs. So NIV helps to address the second roadblock as well.
Of course, NIV isn’t the only way the industry is going to address the need for multi-hop FCoE. Further—to my knowledge, at least—NIV is a Cisco-only approach. As FCoE continues to mature and the IEEE DCB standards are finalized and ratified, then organizations can leverage a standards-based approach to building more complex FCoE topologies than are currently possible today.
Courteous comments, corrections, and thoughts (with full vendor disclosure, please) are welcome below.
Tags: Cisco, FCoE, Networking, Virtualization
-
Scott,
Really like the NIV twist to FCoE. never thought of it like that.
I wanted to point out that this is not technically correct:
“Second, there was no defined standard for creating the FCoE equivalent of trunking E_ports (which I believe are referred to as VE_ports). VE_ports are necessary to link multiple FCFs together, much in the same way you would use an inter-switch link (ISL) in traditional Fibre Channel storage area networks (SANs). To my knowledge, this issue has not yet been addressed.”
VE_Port to VE_Port (FCF to FCF) is included in the current standard FF-BB-5 within the FC-BB_E (7.8.4.2 & 7.8.7.4.4).
The VE_port to VE_Port connection is established via FIP ELP (Exchange Link Parameters / SW_ILS) and not FCoE. So as you mentioned this will only work with the Gen 2 CNAs which include FIP. The major reason for defining a VE_Port is for a logical ISL to be built. Cisco has a couple diagrams on their site illustrating VE_Port to VE_Port with FIP.
However, as of yet the Nexus 50×0 or UCS 62×0 do not include support for it within NX-OS 4.1(3)N1(1). I would bet that Cisco waiting for the IEEE DCB to get finalized may be one part (blog post on that here: http://nickapedia.com/2010/03/19/fcoe-multi-hop-why-wait/). I would also bet Cisco implementing better (up t their standards) FIP-related control would be the other.
So none of the current hardware will take advantage of the definition. But within FIP they do exist in the standard.
Thanks,
.nick
*disclaimer – I work for EMC as a rookie on the same team as Scott Lowe -
Nick,
It’s interesting that you say that VE_ports are included in the FC-BB-5 standard. Not that I doubt you (I haven’t read the standard myself, although it looks like I need to now), but in my discussions with Cisco SAVBU last summer (well after FC-BB-5 was finalized) they intimated to me that the tagging issue—about how to denote VSANs/VLANs across a VE_port—had not yet been addressed. Perhaps this is distinct and separate from the actual creation of VE_ports and thus is a Cisco-specific limitation?
As for the idea of tying NIV and FCoE together, I can’t take the credit. It was actually Brad Hedlund that got me thinking about it last year during a series of e-mail exchanges. It took me until now to be able to articulate NIV in a way that people can understand and then link it back to FCoE.
Thanks for the additional information, Nick!
-
Scott,
Nice post! I love the “what is a hop?” question. That always stirs up a healthy discussion
Cheers,
Brad -
How about another twist on the multi-hop FCoE story….
Instead of doing FCoE forwarding using VE_Ports, how about doing it via NPV?
For instance, utilizing VE_Ports to create a virtual ISL:
VN_port (host) -> VF_port (switch#1) -> VE_port (switch#1) -> VE_port (switch#2) -> VN_port (storage)
So you’d have the host talking to a switch#1 and switch#1 would have a VE_port talking to a VE_port on switch#2 (here is the virtual ISL) and switch#2 would have a storage array attached to it (assuming native FCoE connection here).
In the NPV case….
VN_port (host) -> VF_port (switch#1) -> VNP_Port (switch#2) -> VF_port (switch#2) -> VN_port (storage)
Here we have the host connected to switch #1, however switch#1 and switch#2 talk using VNP_port/VF_Port combo (NPV/NPIV) instead of the virtual ISL.
Instead of needing an FIP ELP, it could be done via FIP FDISC instead.
Thoughts?
-
Scott / Vijay,
There’s another approach that does not require VE_Ports or NPV.
Switch #1 could be a standard DCB capable switch that simply forwards the FIP upstream from the Host to Switch #2. In this case the “hop” is simply an ‘oE’ hop. The host is logging in to Switch #2, where the Storage is also connected.
Physical Ethernet Topology:
Host –> Switch #1 –> Switch #2 –> StorageLogical FC Topology:
Host ——————-> Switch #2 –> StorageSwitch #2 is most Ethernet designs today is a modular chassis based switch.
Nick,
I submit to you that widespread “FCoE Muti-hop” has nothing to do with “waiting for IEEE DCB standards”. The standards are already there. Rather, the gating factor will be when highly scalable modular Ethernet switches (Switch #2) such as Nexus 7000 start shipping DCB capable linecards and FC capable software.Cheers,
Brad
(Cisco, Data Center Architect)Lo
-
Brad, in the scenario you laid forth, Switch#1 would need to support FIP snooping correct? Is the above possible with Nexus 5K or any other switches today?
It seems to me the culprit for all the “FCoE” confusion is what the delta between what is in the standard, versus what is actually possible to do with current HW/firmware TODAY.
-
Vijay,
FIP snooping is the mechanism the Nexus 4000 uses today, if my understanding is correct.
-
Vijay,
Switch #1 doesn’t *need* FIP Snooping, but it is recommended for additional security. The Nexus 5000 in it’s current form does not support FIP Snooping.
Scott,
You are correct. The Nexus 4000 shipping today does provide FIP Snooping and plays the role of a DCB capable switch (Switch #1).
Cheers,
Brad -
Brad,
My DCB argument is that the FC-BB_E requires *lossless* ethernet to run the necessary classes. Since DCB is what is going to turn Ethernet into a proper lossless design in my mind it is a requirement for truly using FCoE as a common protocol (multi-hop)
Here is a quote from the FC-BB_E section:
“The protocol mapping defined by FC-BB_E is referred to as Fibre Channel over Ethernet (FCoE) and requires the underlying Ethernet layer to be full duplex and lossless (i.e., to be composed only of full duplex links and to provide a lossless behavior when transporting FCoE frames).”I am strictly speaking to the standards and not the Vendor implementations (which aren’t there yet). In any network once you start to incur multiple hops you run into more an more congestion opportunities (aggregation, lack of control). The standard is written with VE_Ports defined and lossless requirement. Which explains the large focus of PFC in DCB by IEEE.
Nick
-
Nick,
If you are saying that Vendor implementations of VE_Ports are not there yet, I agree with you.
If you are saying that DCB standards are not there yet to provide “FCoE multi-hop”, I do not agree with that. FC-BB-5 specifies FIP, and the DCB standards for PFC and ETS have been finalized to the point of creating hardware for some time, any additional progress needed at this point is mostly setting meetings and rubber stamping.
I’m highly skeptical that implementations of VE_Ports will be the gating factor to enabling broader adoption of FCoE “multi-hop” anyway. The VE_port implies a point of FC management and a Domain ID, two things that work against scaling the FC network.
Ethernet NPV (as Vijay described), or standard ‘oE’ hops over DCB capable switches using FIP will be the predominant methods. Neither of which introduce a Domain ID into the FC network.
Combine those two approaches with FCoE riding over NIV & Fabric Extenders at the links closest to the Server, as Scott has described here, and you have a highly scalable FCoE “multi-hop” environment without a bunch of VE_Ports.
Just my humble opinions here, nothing more.
Cheers,
Brad -
Fair enough, though my only comment was the VE_port exists in the standard. I didn’t comment on mutli-hop above the Ethernet layer or VE_ports use. That was the other guy. And to be honest I kind of like the NIV idea and I definitely love the oE idea in a DCB network.
Just to be clear, DCB is *not* finalized and other than the Nexus 5k (maybe Brocade 8000 also?) I don’t think there is another DCB capable FCF out yet from a major vendor. But, I haven’t done a lot of research and I am *not* an authority.
Unless the 7k have DCB line cards now. I might have missed them coming out. I knew it would be sooner or later. When that happen it would bring a multi-tier DCB capable network (TOR & dist & core at least). Because we can’t build a network out of just 5k’s.
And just to reiterate, I am NOT commenting on the FCoE or FIP frame not being multi-hop capable. I keep getting dragged into that one. But there is no fricking way someone is going to put production data on a multi-hop/multi-tier FCoE network without DCB across all the bridging devices. So maybe that is only a week away but it isn’t today that I am aware.
I checked here:(http://www.cisco.com/en/US/solutions/collateral/ns340/ns517/ns224/ns783/qa_c67-461717.html) and it is stating that 7ks are still waiting but confirms 5k are ready now.
Good times
Nick
-
Nick,
Of course any switch or link carrying FCoE should be DCB capable. If that’s what you were trying to say I am in full agreement.
My point about the IEEE DCB standards with regards to FCoE multi hop is that no further development is needed with the standards that are actually required for FCoE (PTS, ETS, DCBX). There are other standard efforts lumped into IEEE DCB, such as L2 multi-pathing, which has nothing to do with carrying FCoE. Because the IEEE DCB L2 multi-pathing standard (TRILL) is further behind than others, some people tend to make a generic characterization that “DCB is not ready” in the context of an FCoE conversation, which is both inaccurate and misleading.
Cheers,
Brad -
Err .. meant to say “PFC” as one of the required DCB standards for FCoE, not “PTS”.
I’ve got too much Cisco UCS on my mind these days. PTS is a capability in UCS called “Pass-through Switching”
-
FCoE Multihop CNA connecticity is in fact CNA VN mutlihop access to FCF VF so in this example we have FCF forwarder that serves VF port which is not directly attached to the Server CNA or CNA NIV but is a accessible across a few DCB bridges (FIP Snooping bridges – which do not and can not server VF port function – they have no FC switching logic integrated).
-
to be correct -FCoE multihop is enabled with VE-to-VE FCoE link on the top of DCB connection beteween two Nexus switches or Nexus to MDS switch.
whereas VN to VF FCoE link using FIP snopping mode, is called remote VN-VF connection.
Site Sponsors
15 comments
Comments feed for this article
Trackback link: http://blog.scottlowe.org/2010/03/21/how-niv-and-fcoe-play-together/trackback/