Why No Multi-Hop FCoE?

Much ado has been made—some of it by yours truly—about the current lack of ability to create a multi-hop Fibre Channel over Ethernet (FCoE) fabric. After digging in deeper with Cisco during my recent Unified Computing System (UCS) class, I have some additional information to share about the different forms of multi-hop FCoE and why multi-hop FCoE still isn’t available.

Multi-hop FCoE falls into two basic scenarios:

  • FCoE initiators and/or FCoE targets separated from an FCF (fibre channel forwarder) by multiple hops through IEEE DCB-compliant Ethernet switches
  • Multiple FCFs chained together to connect FCoE initiators and FCoE targets

There are additional scenarios, but for now let’s discuss just these two.

In the first scenario, FCoE initiators and FCoE targets might be separated from an FCF by one or more IEEE DCB-compliant Ethernet switches (also known as an “FCoE passthrough”). In this situation, FCoE Initialization Protocol (FIP) would be required in order for the FCoE initiators and FCoE targets to communicate. Now that FIP support is beginning to emerge following the ratification of the FC-BB-5 standard in early June, this sort of scenario becomes more possible.

If you think like me (and if you do, I’m very sorry to hear that!), your next question is, “OK, what is an IEEE DCB-capable switch?” As it turns out, the Nexus 5000 can be an IEEE DCB-capable switch (or an FCoE passthrough). Cisco doesn’t advertise that fact because they don’t feel that building a solution out of a bunch of Nexus 5000 switches is the best approach. OK, fair enough, so the Nexus 5000 isn’t really designed to be used that way. So what other options are there? None of which I’m aware, at this point, so that makes it impossible to build multi-hop FCoE solutions today. When a valid IEEE DCB-capable switch or FCoE forwarder does appear then you’ll be able to build these sorts of designs—assuming that you have FIP support in both the FCoE initiators and the targets. (Note that you could mix pre-FIP components in here, but all such components would have to be connected directly to the FCF, and would only be able to communicate with other components connected directly to the FCF.)

In the second scenario, FCoE initiators and targets are connected directly to an FCF—like a Nexus 5000—but you’ve got multiple FCFs chained together to create a larger fabric. You might consider this analogous to linking multiple MDS 9000 series switches together with inter-switch links (ISLs). In this case, FIP support would still be necessary for initiators to connect to targets on a different FCF, but now there’s another wrinkle. You see, Cisco has the concept of a VSAN (think of it like a VLAN for Fibre Channel—this is a simplistic definition but reasonable enough to use). In the MDS world (keeping in mind that NX-OS, the software running on Nexus switches, has its roots in SAN-OS, the software that runs on MDS Fibre Channel switches), there is the concept of trunking E_ports, where multiple VSANs are carried on a single E_port between two MDS switches. Continuing the VSAN/VLAN analogy, a trunking E_port is analogous to an 802.1q VLAN trunk.

Bear with me, there’s a reason I’m telling you all this.

When you use FCoE on a Nexus 5000, you end up mapping each VSAN to a VLAN. When you need to move from one FCF to another FCF—i.e., from one Nexus 5000 to another Nexus 5000—how should the VSAN information be presented? Should the VSAN information reside in the 802.1q VLAN tag, so that an 802.1q VLAN trunk is considered a trunking E_port with regard to VSANs? Or should the VSAN information remain embedded in the FC commands that are encapsulated by Ethernet? This fundamental question has not yet been answered. There are advantages and disadvantages to each approach, and as the T.11 group responsible for FC-BB-5 and other FCoE standards hasn’t yet come to agreement yet on how to handle this, then it’s currently not possible to create the FCoE equivalent of trunking E_ports (I believe these will be referred to as VE_ports). Since you can’t create VE_ports, you can’t connect multiple FCFs together, and you can’t build a multi-hop FCoE fabric composed of multiple FCFs.

As you can see, then, that even with FIP present in all components, neither definition of multi-hop FCoE is possible today. Although a Nexus 5000 can function as an FCoE passthrough, Cisco doesn’t recommend that architecture. Without any other IEEE DCB-capable Ethernet switches available to use as an FCoE passthrough, that makes the first scenario impossible to build. Likewise, the inability to create VE_ports and trunk VSANs across multiple Nexus 5000 switches means that it’s impossible to build the second scenario today. While multi-hop FCoE is the ultimate goal, it’s just not possible right now.

Here’s some food for thought while you digest this information: how would a fabric extender change things? That’s a topic I’ll delve into in a future post, so stay tuned!

Of course, FCoE experts and wizards are encouraged to add your corrections, clarifications, and thoughts in the comments below.

Tags: , , , , ,

17 comments

  1. Ronny’s avatar

    And what’s now exactly the advantage of the Nexus 5000? I thought that the “convergence” of all the different technologies (like FC, FCoE and Ethernet) on a single Switch and the cost savings out of this approach was their argument?

  2. David Magda’s avatar

    That’s what I’ve been wondering. What problem is FCoE supposed to solving? If we have to buy special cards and switches, how is it different than plain FC? If I could use the on-board NICs and 4500s/6500s I currently have (perhaps using a separate VLAN), then sure, I have less infrastructure to worry about.

    If I really want bandwidth I could go with iSCSI over InfiniBand at 24, 48, or 96 Gb/s. Probably get lower latency as well.

    Scott, you asked this question in December. Did any of the answers satisfy you, or did you perhaps find enlightenment elsewhere?

    http://blog.scottlowe.org/2008/12/09/continuing-the-fcoe-discussion/

  3. slowe’s avatar

    David,

    FCoE is fine as long as you understand its limitations. First, FCoE is (today) an edge technology only; as I’ve stated on multiple occasions, you can’t really build an end-to-end FCoE solution of any substantial size. Not anytime soon, anyway. As an edge technology, its key redeeming factor is the fact that it reduces and simplifies the connections to servers at the edge of the network.

    Second, FCoE is really only a good option for shops with an existing FC fabric. Because it is completely compatible with “traditional” FC, it’s reasonably easy to extend your existing FC fabric with FCoE at the edge, leveraging the investment you have in FC gear. If you aren’t already using FC, then there’s no need to go FCoE—just use iSCSI or NFS over 10GbE and be done with it.

    I’m not sold on FCoE as the “be all end all” that so many vendors (cough, cough, Cisco) portray it to be. It’s a reasonable edge convergence solution for shops with significant FC investments. For everyone else, 10GbE will address the majority of their needs.

    Does that make sense to you?

  4. slowe’s avatar

    Ronny,

    My response to you is similar to my response to David—the advantage of the Nexus 5000 is cable/connection/port consolidation at the edge of the network for organizations that have existing investments in FC gear. If that’s not you, then the Nexus 5000 is valuable to you only as a good 10GbE switch. Unless you’re small enough that the Nexus 5000 can be your whole solution, that is.

    Does that answer your question?

  5. Ronny’s avatar

    yeah Scott absolutely!

    thanks

  6. Louis Gray (Emulex)’s avatar

    Hi Scott,

    Here are some thoughts on your blog from our technical team here at Emulex.

    The blog seems in many cases to struggle with the gaps of the shipping products and the completeness of the standard. We can’t speak for the current capabilities of Cisco products, but since the standard is mostly written by Cisco, what is standard seems like a good indication of what they will soon offer. To clarify what the standard supports:

    1. We don’t believe FC-BB-5 restricts the types of network configurations provided as examples. It is all about the timing of product introduction (i.e., DCB compliant Ethernet switches) and implementing functionality specified by FC-BB-5 (e.g., “FCoE aware” DCB switches that create/remove ACLs to emulate the security of FC point-to-point links and FCF VE_Port to VE_Port connectivity).

    2. We think that the limitation of having to go through the FCF for native FCoE Initiator to FCoE Target communication in FC-BB-5 has the potential for being a future issue, but work in FC-BB-6 appears to address this. “FCoE aware” DCB switch firmware would probably have to be enhanced to support FCoE virtual link security for direct communication between native FCoE devices.

    3. We don’t think it’s a limitation in the current environment. Our understanding of the initial deployments is that the FCFs will be at the top of rack. If you have 10G Ethernet pass through modules in the blade chassis’, you shouldn’t have any intervening Ethernet switches.

    4. “FCoE end devices cannot access an FCF via multi-hop paths”: Given FIP, the standard fully specifies how FCoE end devices can access an FCF via multi-hop paths. (In fact, the standard would have difficulty preventing it…even without FIP, a preconfigured FCF address would suffice.). DCB-capable intermediate switches are presumed by the standard, though it does not in fact require them..

    5. “FCFs cannot access FCFs via multi-hop paths”: The is still some standardization work to be done on how to map VSANs between a Fibre Channel VSAN header and an Ethernet VLAN header. The standard strongly encourages implementations to use ONLY VLAN headers, so an implementer would not need to do the mapping. The FCFs, in fact, can prevent end devices from using the VSAN header, so the issue need not arise. If an unfortunate installation happened to mix FCFs that required VSAN headers with FCFs that supported only VLANs, they would probably have a problem. But FC-BB-5, together with other Fibre Channel standards, provides enough information to enable an FCF to map between the two methods.

    6. “No Such Thing as an End-to-End FCoE Solution”: The standard fully supports end-to-end FCoE and some of the recent announcements from NetApp and Brocade have addressed some of the end-to-end questions and other storage vendors will follow…The standard fully supports end-to-end FCoE.

  7. slowe’s avatar

    Louis,

    Thanks for your comments.

    What I’m trying to do here is point out the gap between the standards and what’s actually available today. Customers are out there buying FCoE-based solutions with what is, in my opinion, an incomplete view of what is actually available. I’m not talking about what’s possible—I’m talking about what’s available. The vendors give people plenty of information on what’s possible, but what’s lacking is plain speaking on what is actually available.

    Now, I’ll grant that I am not familiar with Brocade’s product line and therefore I’ll admit that some of these “gaps” may exist with Cisco’s product lines but not with other vendor’s product lines.

    1. FC-BB-5 doesn’t restrict the types of network configurations, AFAIK. I provided two examples—certainly there could be many other configurations. The key point here, and one that you support, is that there are NO IEEE DCB-capable Ethernet switches available today. Therefore, customers attempting to build FCoE-based solutions today cannot build solutions architected on IEEE DCB-capable Ethernet switches connecting to an FCF. All the pieces aren’t there.

    2. Once again, that’s all well and good, but all the vendors are just now getting around to full support for FC-BB-5, much less the as-yet-unratified FC-BB-6. Again, my beef is not with the standard, but with the real implementations available to customers.

    3. You don’t think what will be a limitation? The lack of multi-hop support? I agree, it probably won’t be an issue. But the fact remains that many customers don’t know about this limitation, and I think it’s only fair to point this out so that they can plan accordingly.

    4. That statement is made in the context of the lack of availability of IEEE DCB-capable Ethernet switches. Is there such a device that a customer can order today? If there is, let me know so that I can point it out to my readers. If not, then I stand by my earlier statement—if the components aren’t available today, then customers can’t buy and build these solutions today.

    5. Again, this statement is made based on today’s available functionality. What other FCFs are available besides the Nexus 5000? Will those FCFs, today, create VE_ports between them? If so, then point these products out to me so that I can bring them to the attention of my readers. Otherwise, I will again stand by this statement—this is not an arrangement that customers can buy or build today.

    6. Go back and read my corrected article and you’ll see that I retracted that statement. Certainly you can build end-to-end FCoE solutions on a single FCF today. Anything more than that just isn’t possible today, unless you know of a) FCFs that will create VE_ports between them; or b) IEEE DCB-capable Ethernet switches that will act as FCoE passthrough devices.

    Again, let me re-iterate: my purpose here is to educate people on the CURRENT STATUS of actual, available FCoE products. We’re not talking about future standards, future developments, or even what current standards support that no one has bothered to implement.

    Finally, I will again echo my statements regarding FCoE’s usefulness as an edge convergence technology—this use case is possible today, products are available for purchase today, and it has real value in customer’s data centers today. Selling/pushing/supporting FCoE on that basis is fine. Selling/pushing/supporting FCoE for more than that is, in my mind, a disservice to customers given the current state of products and standards implementations.

  8. Eric’s avatar

    Quick position: Today FCoE is top-of-rack gateway to FC and conventional LAN – focused on cable management solution for blades/rackable servers.
    Much ‘ethernet’ work still developing for more complex CEE networks.

  9. slowe’s avatar

    Eric,

    Exactly! If everyone were positioning FCoE in this way—which is really where it is best suited given the current generation of products actually available to customers—then things would make a lot more sense.

  10. Brad Hedlund’s avatar

    Scott,

    You said: “Selling/pushing/supporting FCoE for more than that is, in my mind, a disservice to customers…”

    My comment to that is this: Articulating what the future holds with a technology investment and how it can transform something as significant as the Data Center is what customers expect from their trusted vendors.

    A fast food restaurant only sells and discusses what is currently available, with no discussion of futures, investments, and return on thereof. There is no value in the relationship you have with your local fast food restaurant.

    A technology vendor such as Cisco or Emulex, on the other hand, is never just selling what is available today. Rather, such vendors are selling a long term investment in technology architectures that affect planning with budgets and significant spending decisions. Customers want to make the best possible decisions with their long term investments, and discussing only what’s on the menu today provides no value to that effect.

    Cheers,
    Brad

    (vendor disclosure: Cisco Systems)
    (disclaimer: I am not an official Cisco spokesperson)

  11. slowe’s avatar

    Brad,

    You and I both know that there is a difference between educating/positioning a customer on technology trends so that the customer is prepared, and overselling a technology. Certainly we want our customers to be knowledgeable about where the industry is heading, and to plan and position themselves so that they can take advantage of technology for the benefit of their business. Recommending that a customer deploy FCoE today as an edge convergence technology with an eye toward a complete FCoE implementation/Unified Fabric in future years—as long as the customer understands the limitations in place today—is great. Implying, stating, or otherwise allowing a customer to believe that FCoE can do more than it actually can do, based on the state of today’s available’s products and implementations, is wrong. Let’s not confuse the two.

  12. RS’s avatar

    Great job, Scott. I am intimately familiar with the ins and outs of FCoE (I work for one of the major vendors in the space). You have done a great job of setting real world, level expectations here. You are right about the problems presented in multi-hopping FCoE. These problems are compounded even more by the limitations of spanning tree in Ethernet networks. Looking forward to the now and the next year, with multi-path capable Ethernet fabrics becoming available, this becomes a non-issue, and multi hopping becomes a realistic proposition.

    I have been presenting talks on FCoE for the past few years, and I can say wholeheartedly, there are a lot of misinformation and unrealistic expectations floating around in the industry. We as technology evangelists, whether we work for the vendors or not, owe the community the honesty and candor it deserves.

    Your last point on the “difference between educating/positioning a customer on technology trends so that the customer is prepared, and overselling a technology” really hits the mark here.

    Thanks again! Keep up the good work.

  13. Jit’s avatar

    So Scott, After 2 years do you have any update on this. I mean, I can see that Cisco supports end-to-end FCoE today and with Multihop as well. But VCE is still not supporting in their Vblock. Any insights as to what is the rationale behind that?

  14. slowe’s avatar

    Jit, multi-hop FCoE on Cisco UCS is—to the best of my knowledge—still not supported today. Multi-hop FCoE outside of UCS is supported. As for the rationale or reasoning, you’d have to talk to Cisco for that. Thanks for your comment!

  15. Raj’s avatar

    Hi Scott.

    Great document. Nicely written.

    We are running into the same issue. We are trying to do dual hop fcoe solution with HP VC 10/10D. We are running N5K in NPV mode, since the zoning is done on the upstream brocade switches. Now HP says we have to run the N5K in FCF mode and not NPV, which isnt possible in our scenario (since we dont have MDS as director).

    Would you have any pointers on this ? Would this work or not ? Can we run N5K to support both the existing FC connections, and the new FCOe connection to HP VC ?

Comments are now closed.