Potential UCS Issue: Northbound FCoE Connectivity

I’m about halfway through the first day of Unified Computing System (UCS) training in San Jose, CA, and I’ve learned of what I think is a fairly significant limitation. The issue centers around what Cisco refers to as “northbound” traffic and how Fibre Channel over Ethernet (FCoE) is handled with northbound traffic.

Recall that a central part of UCS is the UCS 6100 series fabric interconnect. The 6100 series fabric interconnect has connectivity in two directions:

  • Southbound connectivity is connectivity aimed back at the fabric extenders in the blade chassis themselves.
  • Northbound connectivity is connectivity headed outside the UCS to other systems and networks.

All southbound traffic is 10Gbps Ethernet with FCoE. Northbound traffic can be 10Gbps Ethernet or Fibre Channel, but not FCoE. Based on the information I’ve been given (and if I’m incorrect please let me know in the comments), you cannot directly connect an FCoE-enabled storage array to a UCS. Even if your storage array has native FCoE interfaces, you can’t plug them into the UCS 6100 series fabric interconnects because that’s considered northbound traffic and you can’t use FCoE with northbound traffic.

I have a feeling customers who have purchased storage arrays with FCoE interfaces with the intention of hooking the arrays up directly to a UCS are going to be a bit upset when this information becomes more widely known.

If I’m working from incorrect or incomplete information, please feel free to speak up in the comments.

Tags: , , , , ,

  1. Rick@VMwareTips’s avatar

    Extremely surprising to hear, I’ve verified that the Cisco website shows the 6-port 10GbE expansion module but does not show any FCoE support. Surprising because the Nexus 5000 has a similar 6-port 10GbE expansion module that shows full FCoE support.

  2. Brad Hedlund’s avatar

    Scott,
    FCoE “northbound” from UCS is not a hardware limitation, and as such will be possible with future software updates now that FCoE and FIP is fully ratified.

    Cheers,
    Brad

  3. Russell’s avatar

    Ask if you can plug your array right into the 6100 to have support. That seems like a reasonable compromise. (I meant to ask this question when I was in training)

  4. Tom Howarth’s avatar

    The biggest issue I have is that this is total vendor tie in, a UCS is useless without a Nexus 5k or 7K. competitors Blade chassis are fabric neutral.

  5. Brad Hedlund’s avatar

    Tom,
    Not true. UCS can be connected to any upstream 10G switch, and any upstream FC SAN switch – even something non-Cisco.

    Cheers,
    Brad

  6. slowe’s avatar

    Russell,

    You can’t for FCoE; I did ask that. I’ll find out about plugging in directly for Ethernet or Fibre Channel.

    Tom,

    That’s actually incorrect. You could uplink Ethernet or Fibre Channel out of UCS (i.e., northbound) to any standard Ethernet or standard Fibre Channel equipment. Obviously, Cisco would prefer that you uplink to a N7K or a N5K, but that’s not required.

  7. rodos’s avatar

    I understood this to be a software issue too.

    This is similar to the 242 vLANs per UCSM which is set to be updated in the next software release.

    Keep the good stuff coming.

  8. Brad Hedlund’s avatar

    Russell,
    The current UCS Manager software operates the FC functions in NPV mode, not FC switch mode. Therefore all zoning and FC forwarding decisions are made by the upstream SAN switch. When UCS software supports FC switch mode you will be able to connect an FC storage array directly into the UCS Manager.

    Cheers,
    Brad

  9. Russell’s avatar

    Thanks for the clarification Brad! I actually remember something along those lines when I was in the training but I had forgotten. I should refer to my notes more frequently.

    Regarding the whole “you gotta have a nexus!” thing; Cisco made it very clear in my UCS training that they don’t care if your network upstream is Brocade/Foundary or even HP Procurve with a QLogic SAN. The 6100 hands off standard 10Gbe and standard FCP. I think the idea is that a lot of people will be plugging these in to Cat 6500s and MDS switches.

  10. Dave Convery’s avatar

    So, does anyone know the timeframe for the software update to allow northbound FCoE?
    Dave

  11. gnijs’s avatar

    I don’t find this a limitation at all, but an interesting feature. Basically, what you are asking for is that UCS becomes a FC switch. And it is just not that (currently). It is not a fabric switch, it just represents multiple hosts on an N port (using NPIV, i am no FC expert at all…). And honestly, would you really want a Cisco fabric switch in your SAN if you are standarized on Brocade or something else ? i guess not, so it keeps the flexibility to connect ucs to no matter what. The same goes for the LAN side, besides. You don’t need a nexus at all, this comment is a little “biased”

  12. Frank’s avatar

    Could you attach the UCS to a N5k and attach the FC storage to the 5k and run FCoE?

  13. slowe’s avatar

    You could run FC out of the UCS6100 to an FC module in the Nexus 5K, then FCoE to a storage array from there—assuming that setup is supported and validated by all the various vendors.

  14. Bozo Popovic’s avatar

    Hello,

    i have to confirm Scotts worries. I assume with new version that they will allow 10G FCoE all across datacenter. This is abstract from UCS Config guide 1.3

    “Network Type
    The network type is only relevant to traffic on uplink ports, because FCoE does not exist outside Cisco UCS. The rest of the data center network only differentiates between LAN and SAN traffic. Therefore, you do not need to take the network type into consideration when you estimate oversubscription of a fabric interconnect port.”

    and

    “Cisco UCS uses Fibre Channel over Ethernet (FCoE) to carry Fibre Channel and Ethernet traffic on the same physical Ethernet connection between the fabric interconnect and the server. This connection terminates at a
    converged network adapter on the server, and the unified fabric terminates on the uplink ports of the fabric interconnect. On the core network, the LAN and SAN traffic remains separated. Cisco UCS does not require that you implement unified fabric across the data center.”

    With regards.
    Bozo

  15. Bill’s avatar

    Has anyone heard any updates as to when Cisco may begin to offer Northbound FCOE functionality? Even better if it could be offered in Switch mode.

  16. Scott’s avatar

    I’m finding this a pretty severe limitation as well. I’m not understanding the limitation of making an FCoE port as an NP port upstream to an NPIV FCoE switch like the Nexus 5k. Why does the uplink port have to be native FC? I don’t mind not having switching in the interconnects but it would save a lot of money if I could use FCoE uplinks as an NP port to the 5k and have it act as my FC switch and connect to my FC array(s). I have a lot of customers that are looking to replace their 5-6 year old FC switches and I could justify selling N5K’s as replacements.

    Any chance at all this will be released in the update coming in December?

  17. Jean’s avatar

    Hi there,

    is FCoE northbound to a Nexus 5010 Switch still an issue, or is it solved in UCS Version 1.4 3q and upwards.

    Regards
    Jean

  18. Sean’s avatar

    Jean, I do not think it northbound FCoE is here yet. I could not find any blog posts or discussion about it and I would think it would be a pretty big deal which would have seen some discussion. I looked over the public release notes for UCS manager on Cisco’s site all the way up to 2.0 and did not see anything, hoping I missed it somewhere but I am not holding my breath. Time to add a few FC cables to my diagram.

  19. Sameer’s avatar

    UCS does not support FCoE Northbound connectivity yet.

    With a Service Profile, you can only assign regular NICs (10gb), and FC HBA’s, you can’t assign a CNA adapter to a service profile.