Thinking Out Loud: What is multi-hop FCoE?

It’s been a while since I last posted a “Thinking Out Loud” post, but a Twitter discussion this morning prompted me to post this one. Last night, I posted a question to Brad Hedlund (@bradhedlund on Twitter and a great data center networking resource) regarding the Nexus 2232PP fabric extender. My tweet to Brad was this: does the Nexus 2232PP make “multi-hop FCoE” possible via NIV?

(If you haven’t already read my post on the relationship between FCoE and NIV, go read it now as it provides a good background and context for this discussion.)

Brad responded, confirming my assessment and stating that FIP snooping wasn’t necessary. I wasn’t sure about the FIP snooping part but after seeing the response I now understand. In any event, a number of others starting questioning my use of “multi-hop FCoE” in conjunction with the 2232PP fabric extender, stating that it wasn’t a hop because ti does not actually switch any traffic.

Strictly speaking, all of these individuals are absolutely correct; for more information, go read this post on NIV. In this case, the Nexus 5000 is the interface virtualization (IV)-capable bridge and the Nexus 2232PP is the interface virtualizer (IV). The IV doesn’t switch any traffic; all switching occurs on the IV-capable bridge. Therefore, from a switching perspective, a Nexus 5000 and any or all associated fabric extenders are a single hop.

My response is this: what is multi-hop, anyway? Is it the presence of multiple switches in a data path between an initiator and a target? Or is it the presence of multiple physical devices, chained together, between an initiator and a target? In the first definition, using a fabric extender isn’t multi-hop; in the second definition, it is. More to the point, should a customer really care which definition is correct? Why is multi-hop FCoE important, anyway? I would say the answer to that question is scalability. Customers want to be able to scale the FCoE fabrics larger than they can today. Multi-hop FCoE—however you want to define it—makes that possible. So does it really matter how we get there? Besides which point, using the fabric extender approach means only a single point of management instead of multiple points of management. Isn’t that better?

What do you think? Do your own “thinking out loud” in the comments below.

Tags: , , , ,

  1. ev’s avatar

    Interesting question. RFC1812 clearly treats (although does not define) the word “hop” as either the “destination hop” or the “next-hop” router…

    So does a singler router+destination = > 1 hop = “multi-hop” ?

    Perhaps there is an earlier RFC that defines the term? If not, then it’s a linguistic construct, and its generally accepted definition is just that…

    Thanks for tips.

  2. Joe Onisick’s avatar

    Scott,

    From my perspective the only reason the semantics of hop are important in regards to a 2K is vendor FUD. There is a vendor who constantly throws the 2 hop argument at the 2K/IOM in order to create end user doubt causing them to think of complexity and latency. The reality is it doesn’t matter if the 2K is 0 hops or 10 hops as long as the total switching latency is low enough (it is.)

    For many SANs the single layer design with 2232s will be a great way to avoid complexity and management costs. Two reasons for multi-hop will be core edge designs with director class cores, and SANs requiring SAN extension through FCIP which would require hopping to an FCIP bridge capable switch.

    Overall I agree with you I’d rather use the fabric extender technology as much as possible and minimize management points.

  3. slowe’s avatar

    Joe,

    Vendor FUD…well, that’s something we all have to deal with in some form or fashion. :-)

  4. Joe Onisick’s avatar

    True, and for full disclosur sake I do drink the Cisco Koolaid ;-)

  5. scott owens’s avatar

    multi-hop means source in one data center and consumer in another.
    Or at least that is what I (we) am hoping or waiting for.

    From some of my other posts I think we may be better off with iSCSI which is both multi-hop literate and license free.

    for us.
    SiteA has disks_1 and disk_backup_2
    SiteB has disks_2 and disk_backup_1

    This way our backups are kept at the opposite site and we can currently only do this with CWDM – although that is limited to 2 Gb GBICs – or DWDM/EWDM. If we could do FCOE we could – I think – run both our storage and IP traffic and know the storage gets its fair share of traffic.

    Joe,

    Have you gone through DR/failure testing of your 5Ks & FEXs ? And if you don’t mind are you hooking them up in dual-homed fashion or 1 FEX per 5K ?

  6. scott owens’s avatar

    In my last sentence I meant that each FEX ( however many you have )gets only connected to a single 5K.

  7. Brad Hedlund’s avatar

    Scott,
    Thanks for the nice post. You rock dude!

    Cheers,
    Brad

  8. slowe’s avatar

    Brad,

    No need to thank me—thank you for all the help in understanding how these technologies come together!

  9. Bobby Hyam’s avatar

    I think it all depends on which layer we’re looking at it from. From a layer one (physical) we have multiple physical devices and cables joined together so we definitely have multiple hops and each cable / device definitely adds latency in some shape or form. Note though that the twinax cables add a ridiculously low amount of latency compared to regular copper.

    At layer three (network), obviously it’s a single hop as there’s no routing going on at all.

    At layer two you (data link) have the more interesting question. The ethernet frames are not getting switched by the fabric extender – they’re just getting aggregated. Do the frames actually get modified at this point at all with QoS or VLAN tagging information? I’m guessing there might be some sort of tagging or modification also for the fabric assigned WWNs and MAC addresses?

    Does modification of the frame even constitute a hop? What does?