SAN Port Channels from Nexus 5010 to MDS 9134

As part of an ongoing effort to expand the functionality of the vSpecialist lab in the EMC RTP facility, we recently added a pair of Cisco MDS 9134 Fibre Channel switches. These Fibre Channel switches are connected to a pair of Cisco Nexus 5010 switches, which handle Unified Fabric connections from a collection of CNA-equipped servers. To connect the Nexus switches to the MDS switches, we used SAN port channels to bond multiple Fibre Channel interfaces together for both redundancy and increased aggregate throughput. Here is how to configure SAN port channels to connect a Cisco Nexus switch to a Cisco MDS switch.

If you are interested, more in-depth information can be found here on Cisco’s web site.

Although I’ve broken out the configuration for the MDS and the Nexus into separate sections, the commands are very similar. In my situation, the MDS 9134 was running NX-OS 5.0(1a) and the Nexus 5010 was running NX-OS 4.2(1)N1(1).

Configuring the Cisco MDS 9134

To configure the MDS 9134 with a SAN port channel, use the following commands.

First, create the SAN port channel with the interface port-channel command, like this:

mds(config)# interface port-channel 1

You can replace the “1″ at the end of that command with any number from 1 to 256; it’s just the numeric identifier for that SAN port channel. The SAN port channel number does not have to match on both ends.

Once you’ve created the SAN port channel, then add individual interfaces with the channel-group command:

mds(config)# interface fc1/16
mds(config-if)# channel-group 1

The “1″ specified in the channel-group command has to match the number specified in the earlier interface port-channel command. This might seem obvious, but I wanted to point it out nevertheless.

Repeat this process for each interface you want to add to the SAN port channel. In my example, I used two interfaces.

When you add an interface to the SAN port channel, NX-OS reminds you to perform a matching configuration on the switch at the other end, then use the no shutdown command to make the interfaces (and the SAN port channel) active. Let’s look first at the commands for configuring the Nexus, then we’ll examine what it looks like when we bring the SAN port channel online.

Configuring the Cisco Nexus 5010

The commands here are very similar to the MDS 9134. First, you need to create the SAN port channel using the interface san-port-channel command (note the slight difference in commands between the MDS and the Nexus here):

nexus(config)# interface san-port-channel 1

As with the MDS, the number at the end simply serves as a unique identifier for the SAN port channel and can range from 1 to 256.

Then add interfaces to the SAN port channel using the channel-group command:

nexus(config)# interface fc2/1
nexus(config-if)# channel-group 1
nexus(config-if)# interface fc2/2
nexus(config-if)# channel-group 1

As I’ve shown above, simply repeat the process for each interface you want to add to the SAN port channel. As on the MDS, NX-OS reminds you to perform a matching configuration on the opposite end of the link and then issue the no shutdown command.

Bringing Up the SAN Port Channel

Once a matching configuration is performed on both ends, then you can use the no shutdown command (which you can abbreviate to simply no shut) to activate the interfaces and the SAN port channel. After activating the interfaces, a show interface port-channel (on the MDS) or a show interface san-port-channel (on the Nexus) will show you the status of the SAN port channel. Only the first few lines of output are shown below (this output is taken from the Nexus):

nexus# sh int san-port-channel 1
san-port-channel 1 is trunking (Not all VSANs UP on the trunk)
    Hardware is Fibre Channel
    Port WWN is 24:01:00:05:9b:7b:0c:80
    Admin port mode is auto, trunk mode is on
    snmp link state traps are enabled
    Port mode is TE
    Port vsan is 1
    Speed is 4 Gbps
    Trunk vsans (admin allowed and active)  (1)
    Trunk vsans (up)                        ()
    Trunk vsans (isolated)                  ()
    Trunk vsans (initializing)              (1)

A couple of useful pieces of information are available here:

  • First, you can see that the SAN port channel is not fully up; it’s still initializing. This is shown by the “Not all VSANs UP on the trunk” message, as well as by the “Trunk vsans (initializing)” line.
  • Second, you can see the only a single member is up. Note the speed of the SAN port channel is listed as 4 Gbps.
  • Third, note that this is a trunking port, meaning that it could carry multiple VSANs. This is noted by the “Port mode is TE” line as well as the first line of the output (“san-port-channel 1 is trunking”).

As it turns out, I’d cabled the connections wrong; after I fixed the connections and gave the SAN port channel a small amount of time to initialize, the output was different (this output is taken from the MDS):

nexus# sh int port-channel 1
port-channel 1 is trunking
    Hardware is Fibre Channel
    Port WWN is 24:01:00:05:73:a7:72:00
    Admin port mode is auto, trunk mode is on
    snmp link state traps are enabled
    Port mode is TE
    Port vsan is 1
    Speed is 8 Gbps
    Trunk vsans (admin allowed and active)  (1)
    Trunk vsans (up)                        (1)
    Trunk vsans (isolated)                  ()
    Trunk vsans (initializing)              ()

Now you can see that both members of the SAN port channel are active (“Speed is 8 Gbps”) and that all VSANs are trunking across the SAN port channel.

At this point, you are now ready to proceed with creating VSANs, zones, and zonesets. Refer to these articles for more information on MDS zone creation and management via CLI:

New User’s Guide to Configuring Cisco MDS Zones via CLI
New User’s Guide to Managing Cisco MDS Zones via CLI

As always, questions, clarifications, or corrections are welcome—just add them below in the comments. Thanks!

Tags: , , , ,

  1. Andy’s avatar

    You should consider setting those trunks to non-negotiate/Permanent Trunking. If those trunks were to ever have to re-negotiate it could possibly give your storage network links a hiccup.

  2. slowe’s avatar

    Andy, what would the command be to do that? And are you referring to trunking (carrying multiple VSANs) or port channeling?

  3. Andy’s avatar

    The default behavior of ports would be to Automatically negotiate to be a trunk, so if you leave the ports to their default behavior, they’ll establish a trunk, which you can see from your show command. Cisco recommends – “The preferred configuration on the Cisco Nexus 5000 Series switches is that one side of the trunk is set to auto and the other is set to on. ”

    I would disagree and set it to force trunk on both sides. Auto in the cisco world is something you “Auto-Not-Do”

    I believe the command you would use on the physical interfaces AND the port channel is “Switchport trunk mode on”. I don’t have much experience with VSAN’s but the principals would be similar to VLANs imo.

    Once configured to BE a trunk, they shouldn’t negotiate at all.

    http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/configuration/guide/cli_rel_4_0_1a/trnk.html

  4. Darryl’s avatar

    On a separate but relevant note…

    How do you add a port-channel to a VSAN, and does this satisfy the requirement to have the interfaces on both sides of an ISL Trunk or port-channel added to the VSAN?

    Thanks for responding

  5. slowe’s avatar

    Andy, now I understand your comment. You are referring to VSAN trunking. Thanks for the heads-up; I’ll take a look at that in my configuration.

    Darryl, from a purely Cisco perspective you only need to configure your SAN port channels as VSAN trunks (TE ports) and ensure that the VSANs are properly defined on both ends. That should be all that’s required to make it work. Unless I’m not understanding your question?

  6. Darryl’s avatar

    You are understanding my question correctly. During my research i’m seeing outputs of the “show vsan x membership” command listing port-channels in the vsan.

    switch # show vsan 10 membership
    vsan 10 interfaces: fc2/1 fc2/2 fc2/3 fc2/4 port-channel 3

    How did the port-channel get in the port VSAN? I understand the interfaces on both sides of the ISL must be in the VSANs requiring transport, but is this done on an individual interface basis or simply by adding the port-channel interface?

    Thanks for responding, i know i hijacked the conversation.

  7. slowe’s avatar

    Darryl, you would put the port-channel interface (as well as any member interfaces) into the VSAN using the “vsan database” command—that should accomplish what you are seeking.

  8. Darryl’s avatar

    Thanks for keeping the information flowing…

  9. khuanchai’s avatar

    Hello, I have done to port channel all vsan allow by
    trunking mode form MDS 9124 to Nexus 5010 (the storage connect to
    MDS9124, Host connect to Nexus 5010 via FCOE) how i can check on
    Nexus that the port channel configuration working poperly and also
    how can i map the host that connect on Nexus 5010 to storage that
    connect on MDS 9124 please light me . Thank you,

  10. slowe’s avatar

    Khuanchai, the “sho int port-channel X” (or “show int san-port-channel X”) commands will tell you if the SAN port channel is up or not. As for mapping hosts to storage, that involves both zoning (on the SAN switches) as well as storage presentation (on the array). These articles should help with the SAN portion:

    http://blog.scottlowe.org/2009/08/24/new-users-guide-to-configuring-cisco-mds-zones-via-cli/

    http://blog.scottlowe.org/2009/10/20/new-users-guide-to-managing-cisco-mds-zones-via-cli/

    http://blog.scottlowe.org/2010/12/08/using-device-aliases-on-a-cisco-mds/

    Good luck!

  11. jciarlette’s avatar

    I am having an issue trying to add a port to the port channel. I get “port not compatible: [Port allowed vsan list]” I have trunking on, a single vsan in list, and both ports i am trying to add are allowing same vsan. Help!!! please.

  12. Darren Ramsey’s avatar

    Scott,

    Good info here on your site. Have you looked/tested the N5K in NPV mode with san-port-channel trunking (TNP->TF) to a MDS? NPIV and fport-channel-trunk features must be enabled on the MDS, and NPV on the N5K.

    We have been running this config since NX-OS 4.2 released with good results. However we did find several issues that you might want to be aware of. We tested the san-port-channel with IOMETER and simulated failed san PO uplinks. We ran into a 20 second IO loss due to CSCtd54245 that has since been fixed. We also ran into CSCti57443 on the FEX uplink between our 2232 and N5K. Now MPIO/PPVE will compensate for both of these, but not everyone is running PPVE on ESX.

    —————————————————————
    corp-nex1# sho runn int san1

    !Command: show running-config interface san-port-channel 1
    !Time: Thu Jan 6 10:02:31 2011

    version 4.2(1)N2(1)

    interface san-port-channel 1
    channel mode active
    switchport mode NP
    switchport description MDS-9506-03
    switchport trunk mode on
    switchport trunk allowed vsan 121
    switchport trunk allowed vsan add 131
    switchport trunk allowed vsan add 141

    corp-nex1# sho int san1
    san-port-channel 1 is trunking
    Port description is MDS-9506-03
    Hardware is Fibre Channel
    Port WWN is 24:01:00:05:9b:1e:00:00
    Admin port mode is NP, trunk mode is on
    snmp link state traps are enabled
    Port mode is TNP
    Port vsan is 1
    Speed is 24 Gbps
    Trunk vsans (admin allowed and active) (121,131,141)
    Trunk vsans (up) (121,131,141)
    Trunk vsans (isolated) ()
    Trunk vsans (initializing) ()
    5 minute input rate 10145048 bits/sec, 1268131 bytes/sec, 1568 frames/sec
    5 minute output rate 30105328 bits/sec, 3763166 bytes/sec, 2369 frames/sec
    10381843246 frames input, 15075247432540 bytes
    20 discards, 0 errors
    0 CRC, 0 unknown class
    0 too long, 0 too short
    7736628204 frames output, 11765023489340 bytes
    0 discards, 92 errors
    0 input OLS, 0 LRR, 0 NOS, 0 loop inits
    0 output OLS, 0 LRR, 0 NOS, 0 loop inits
    last clearing of “show interface” counters never
    Member[1] : fc2/1
    Member[2] : fc2/2
    Member[3] : fc2/3
    Interface last changed at Mon Jun 21 11:44:46 2010

  13. slowe’s avatar

    Jciarlette, it sounds to me like there are still some differences between the port configurations. I can’t tell you exactly what without more information, so I would just recommend you check the current trunking status, port speed, VSAN membership, allowed VSANs, and other configurations between the two ports. Good luck!

    Darren, I haven’t yet personally tested that configuration, although I think that a fellow EMC’er did over at the Brass Tacks blog:

    http://brasstacksblog.typepad.com/brass-tacks/2010/12/creating-san-port-channels-when-using-npv-mode-on-nexus-5k.html

    Thanks for the heads-up on the connectivity issues you uncovered—I’m sure that other readers will find this very useful!

  14. khuanchai’s avatar

    Hello.
    Thank you for your previosly answer. i got another question. i have nexus 5010 connect with existing configuration VMware host via FCOE port. my question is about, if existing configuration of vmware require multi vlan in one interface. how i can configuration FCOE on nexus to support multi vlan in a FCOE interface

  15. slowe’s avatar

    Khuanchai, have a look at this article and see if it doesn’t provide you with the information you need:

    http://blog.scottlowe.org/2010/12/15/fcoe-through-a-nexus-to-a-mds-attached-storage-array/

    Good luck!

  16. Kumar’s avatar

    Scott – with Nexus 5Ks supporting FC ports, is there even a need to use MDS? Could you use the same N5k switch for both your FC and ethernet traffic. I remember reading that for FC traffic you have to run the 5k in FC-SW mode. Has this changed?

  17. Geir’s avatar

    I am thinking in the same line as Kumar here, but will use a set of 5596UP as dedicated “SAN switches” aggregating SAN port channels from end-of-row 5596UPĀ“s and connecting them to our Hitachi SAN, any thing to consider or limitations? We are moving from a Brocade SAN network to FCoE, Nexus etc when moving to a new office later this year, so it seams nice to skip MDS and just use an extra set of 5596UP as we will have 12 of as end-of-row switches, HP blade chassis with Nexus B22 FEX

  18. dev’s avatar

    hi scott,

    I am facing issue in configuring cisco c240m3 server to nexus5548 switch. the problem i am facing is , flogi database is not showing the host information.

    following is the configuration from nexus 5548 switch

    version 5.0(3)N2(1)
    feature fcoe

    feature telnet
    cfs ipv4 distribute
    feature lldp

    system qos
    service-policy type qos input fcoe-default-in-policy
    service-policy type queuing input fcoe-default-in-policy
    service-policy type queuing output fcoe-default-out-policy
    service-policy type network-qos fcoe-default-nq-policy

    vlan 151
    fcoe vsan 151

    interface vfc1
    bind interface Ethernet1/1
    no shutdown

    interface vfc3
    bind interface Ethernet1/3
    no shutdown
    vsan database
    vsan 151 interface vfc1
    vsan 151 interface vfc3

    interface Ethernet1/1
    switchport mode trunk
    spanning-tree port type edge trunk

    interface Ethernet1/2

    interface Ethernet1/3
    switchport mode trunk
    spanning-tree port type edge trunk

    Nexus5k-A# show flogi database
    No flogi sessions found.

    Nexus5k-A# show interface vfc 3
    vfc3 is trunking (Not all VSANs UP on the trunk)
    Bound interface is Ethernet1/3
    Hardware is Virtual Fibre Channel
    Port WWN is 20:02:00:2a:6a:2e:4d:bf
    Admin port mode is F, trunk mode is on
    snmp link state traps are enabled
    Port mode is TF
    Port vsan is 151
    Trunk vsans (admin allowed and active) (1,151)
    Trunk vsans (up) ()
    Trunk vsans (isolated) ()
    Trunk vsans (initializing) (1,151)
    1 minute input rate 280 bits/sec, 35 bytes/sec, 0 frames/sec
    1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
    17289 frames input, 3181176 bytes
    0 discards, 0 errors
    0 frames output, 0 bytes
    0 discards, 0 errors
    last clearing of “show interface” counters never
    Interface last changed at Thu Aug 8 10:34:28 2013

    I havent done any specific configuration at server vhba side , everything is default.
    Your help is appreciated.

Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>