Fibre Channel over Ethernet (FCoE) is receiving a great deal of attention in the media these days. Fortunately, setting up FCoE on a Nexus 5000 series switch from Cisco isn’t too terribly complicated, so don’t be too concerned about deploying FCoE in your datacenter (assuming it makes sense for your organization). Configuring FCoE basically consists of three major steps:
- Enable FCoE on the switch.
- Map a VSAN for FCoE traffic onto a VLAN.
- Create virtual Fibre Channel interfaces to carry the FCoE traffic.
The first step is incredibly easy. To enable FCoE on the switch, just use this command:
switch(config)# feature fcoe
The next part of the FCoE configuration is mapping a VSAN to a VLAN. What VSAN should you use? Well, if you are connecting to an existing Fibre Channel fabric, perhaps on a Cisco MDS switch, you’ll need to make sure that the VSANs between the Nexus and the MDS are appropriately matched. Otherwise, traffic on one VSAN on the Nexus won’t be able to reach devices on another VSAN on the MDS. If there’s enough demand, I’ll post a quick piece on this step as well.
Note that this FCoE VSAN-to-VLAN mapping is a required step; if you don’t do this, the FCoE side of the interfaces won’t come up (as you’ll see later in this post). Assuming the VSAN is already defined, perform these steps to map the VSAN to a VLAN:
switch(config)# vlan XXX
switch(config-vlan)# fcoe vsan YYY
switch(config-vlan)# exit
Obviously, you’ll want to substitute XXX and YYY for the correct VLAN and VSAN numbers, respectively.
After you’ve enabled FCoE and mapped FCoE VSANs onto VLANs, then you are ready to create virtual Fibre Channel (vfc) interfaces. Each physical Nexus port that will carry FCoE traffic must have a corresponding vfc interface. Generally, you will want to create the vfc interface with the same number as the physical interface, although as far as I know you are not required to do so. It just makes management of the interfaces easier. The commands to create a vfc interface look like this:
switch(config)# interface vfc ZZ
switch(config-if)# bind interface ethernet 1/ZZ
switch(config-if)# no shutdown
switch(config-if)# exit
At this point the vfc interface is created, but it won’t work yet; you’ll need to place it into an VSAN that is mapped to an FCoE enabled VLAN. If you don’t, the show interface vfc <number> command will report this (emphasis mine):
vfc13 is down (VSAN not mapped to an FCoE enabled VLAN)
As I mentioned earlier, if you haven’t mapped the FCoE VSAN onto a VLAN, you won’t be able to fix this problem. If you have mapped the FCoE VSAN onto a VLAN, then you only need to assign the vfc interface to the appropriate VSAN with these commands:
switch(config)# vsan database
switch(config-vsan-db)# vsan <number> interface vfc <number>
switch(config-vsan-db)# exit
At this point, the vfc interface will report up, and you should be able to see the host’s connection information with the show flogi database command.
From this point—assuming that your storage is attached to a traditional Fibre Channel fabric, which is likely to be the case in the near future—you only need to create zones with the WWNs of the FCoE-attached hosts in order to grant them access to the storage. Refer to my posts on creating zones and managing zones on a Cisco MDS for more information on this task.
In my own experience, once FCoE was properly configured on the Nexus 5000 switch, then creating zones and zonesets on the Cisco MDS Fibre Channel switch and creating and masking LUNs on the Fibre Channel-attached storage is very straightforward. This, as has been stated on several previous occasions, is one of the strengths of FCoE: it’s compatibility with existing Fibre Channel installations is outstanding.
Feel free to submit any questions or clarifications in the comments below.
Tags: Cisco, FCoE, FibreChannel, Networking, Nexus, Storage
-
Awesome post Scott!! Perhaps a nice follow up post would discuss a configuration using NPV mode on the Nexus 5000.
Cheers,
Brad -
Good article, Scott. Thanks. I’d be curious to find out how the performance is and if you are getting 80% better efficiency using 10Gb DCE over multiple 1Gb links.
-
Trackback from Lewan Professional Services Blog on Monday, October 26, 2009 at 11:19 am
-
In regards to Brad’s comment about NPV mode. There is a case study in the EMC FCoE Tech book that shows how to configure the Nexus 5000 for NPV mode. We use it to connect the Nexus to non-Cisco FC switches. For more info see http://www.emc.com/collateral/hardware/technical-documentation/h6290-fibre-channel-over-ethernet-techbook.pdf . I’ve heard someone say (complain?) it goes into bleeding-eyeballs detail, so be warned.
Regards, Erik
-
Hey Scott, good post. Where were you last week when my lab gear showed up!
I kid!
One thing I had to do on my Nexus 5k to get my vfc’s to link was actually make the switchports into trunk ports. I assume you did this but left it out of your article? Either that or I missed something obvious and I’m open to suggestion.
Relevant configs:
vlan 2
fcoe vsan 1
name FCoETransportinterface vfc#
bind interface Ethernet1/#
no shutdowninterface Ethernet1/#
description ESX server lab-rp-esx1 host uplink to 1000v
switchport mode trunk
switchport trunk allowed vlan 1-2,201,1000-1001
spanning-tree port type edge trunkIn this case I’m still using native VLAN 1 for some ethernet traffic. I would imagine if this were a windows server I would probably just specify the native VLAN as whatever.
Interesting things of note: I actually didn’t turn on NPV mode and I’m successfully able to merge with a QLogic fabric in my lab and set up zoning on either side and have things work. I thought I had some issues (I screwed up an igroup on my filers) so I tried it in NPV mode. Either way seems pretty straight forward.
Neat thing I like about NXOS and the nexus 5k in general.
I can do things like this:
int eth1/1-10,e1/20-25,e1/30,e1/37
Handy when you want to disable the second path of every ESX host for example to illustrate failover:
int eth1/11,e1/13,e1/15
shutno shut
Also, I dunno if you could do this with IOS but I thought this was awesome:
int po1-16
This was actually more useful on the nexus 1000v where I needed to create a bunch of vPC-HM port channels to uplink my lab servers.
One thing of note that might seem obvious to some is that you can’t use etherchannel on any ethernet interfaces with a vfc assigned.
-
Russell -
According to the Cisco Nexus 5000 Series Switch CLI Software Configuration Guide, you’re correct – you need to put the physical interface into trunk mode:
I don’t find a note about Etherchannel limitation so I’ll pass that tidbit on to the book writer to investigate/add if appropriate.
-
Trackback from uberVU - social comments on Tuesday, October 27, 2009 at 3:26 pm
-
Thanks Scott. Excellent article. I am curious though about how things would work with native FCoE targets, like NetApp’s target for example. In your example you are configuring connectivity for FCoE initiators attached to the Nexus switch to FC targets attached to a traditional FC fabric. My understanding is that NetApp supports FCoE natively by adding a Qlogic target mode CNA to their array controllers which would have a 10G FCoE capable port.
I had assumed that in this configuration the 10G port on the NetApp controller would connect directly to a port on the Nexus switch.
If that is the case then would VLAN/VSAN mapping still be required? If so, would the VSAN exist on a Nexus switch? And along the same line of thinking, would FCoE initiators be zoned to native FCoE targets on a Nexus switch.
Best Regards,
Vic -
In response to Vic’s questions, the answers are yes, yes, and yes.
The whole beauty is that, when in switch mode (but not in NPV mode), the vfc port acts and behaves just like any F-port. Think of it this way, treat
it this way, and you will always know hot to configure it. -
I have recently started working and learning about FCoE. I have an access port routing all of my network traffic on the FCoE switch to vsan 2 on my Cisco Nexus 5010. My Windows clients only work when I specify the VLAN its connected to for my Linux client I must enable vlan tagging as well or I cannot ping any entity in the subnet is this normal do we have to set the VLAN-ID via the client ?
-
This is what I configurated on the Nexus5548p, but the communitication between SW and Server stuck at initializing. No Tunk Vsan up. I can not see the hose information with the sh flogi database command. What is wrong with it? Is there license issue? Thanks.
============================
feature fcoevlan 20
fcoe vsan 10interface vfc11
bind interface Ethernet1/21
switchport trunk allowed vsan 10
no shutdownvsan database
vsan 10 interface vfc11interface Ethernet1/21
description To ESX11 VMHBA1
switchport mode trunk
switchport trunk allowed vlan 1-2,10,20
spanning-tree port type edge trunkSW5548# sh int vfc 11
vfc11 is trunking
Bound interface is Ethernet1/21
Hardware is Virtual Fibre Channel
Port WWN is 20:0a:00:05:73:af:22:3f
Admin port mode is F, trunk mode is on
snmp link state traps are enabled
Port mode is TF
Port vsan is 10
Trunk vsans (admin allowed and active) (10)
Trunk vsans (up) ()
Trunk vsans (isolated) ()
Trunk vsans (initializing) (10)
1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
0 frames input, 0 bytes
0 discards, 0 errors
0 frames output, 0 bytes
0 discards, 0 errors
last clearing of “show interface” counters never
Interface last changed at Thu Feb 24 16:31:26 2011
============================================ -
Hector, it’s my understanding the interfaces must be configured as trunk interfaces in order for FCoE to work. As a result, there might be some OS-specific configurations that are necessary in order for traffic to flow, based on native VLAN assignments and such.
David, one thing that has come up recently is that the Nexus 5500 doesn’t have the necessary CoS/QoS features enabled to support FCoE; the Nexus 5000 had them enabled automatically. I think this article might help you out:
Good luck!
-
Hi,
I’m configuring UCS involving blade servers with Qlogic CNAs behind Nexus 4000 switches connected to Nexus 5000. My storage is attached to MDS 9148 switches.
I followed FCoE configuration guides and every thing worked fine and I made zoning in my MDS 9148 switches using pwwns.
But my servers and storage still don’t see each other though the show zoneset active command states that the interfaces are active:zone name TTSVCN1_P4_BCH1_BL1_ZONE vsan 20
* fcid 0x1b0000 [pwwn 50:05:07:68:01:40:b7:2c]
* fcid 0xa30000 [pwwn 21:00:00:c0:dd:16:92:f1]When I issue the show fcns database command I have the following result:
VSAN 20:
————————————————————————–
FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE
————————————————————————–
0x1b0000 N 50:05:07:68:01:40:b7:2c (IBM) scsi-fcp:target
0x1b0100 N 50:05:07:68:01:10:b7:2c (IBM) scsi-fcp:target
0x1b0200 N 50:05:07:68:01:40:ba:5d (IBM) scsi-fcp:target
0x1b0300 N 50:05:07:68:01:10:ba:5d (IBM) scsi-fcp:target
0x1b0400 N 20:34:00:80:e5:1b:f0:88 (Mylex) scsi-fcp:target
0x1b0500 N 20:32:00:80:e5:1b:e6:a8 (Mylex) scsi-fcp:target
0x1b0600 N 20:33:00:80:e5:1b:e6:a8 (Mylex) scsi-fcp:target
0x1b0700 N 20:35:00:80:e5:1b:f0:88 (Mylex) scsi-fcp:target
0xa30000 N 21:00:00:c0:dd:16:92:f1 (Qlogic)
0xa30001 N 21:00:00:c0:dd:16:93:ad (Qlogic)
0xa30002 N 21:00:00:c0:dd:16:94:95 (Qlogic)
0xa30003 N 21:00:00:c0:dd:16:92:e1 (Qlogic)
0xa30004 N 21:00:00:c0:dd:16:ae:19 (Qlogic)
0xa30005 N 21:00:00:c0:dd:16:af:01 (Qlogic)
0xa30006 N 21:00:00:c0:dd:16:af:2d (Qlogic)
0xa30007 N 21:00:00:c0:dd:16:93:a5 (Qlogic)
0xa30008 N 21:00:00:c0:dd:16:b5:3d (Qlogic)
0xa30009 N 21:00:00:c0:dd:16:93:e9 (Qlogic)
0xa3000a N 21:00:00:c0:dd:16:af:a1 (Qlogic)
0xa3000b N 21:00:00:c0:dd:16:93:9d (Qlogic)
0xa3000c N 21:00:00:c0:dd:16:ae:39 (Qlogic)
0xa3000d N 21:00:00:c0:dd:16:90:4d (Qlogic)Total number of entries = 22
I’m wondering why the FC4-TYPE:FEATURE of my Qlogic interfaces is blank and if it’s related to the issue!
Do you have any troubleshooting suggestions?
Best regards!
-
Meryem, I’m a bit confused about your setup, as the Nexus 4000 is an IBM-only part that is not used at all in the UCS. Perhaps you have your equipment names incorrect? Without a clearer picture of what sorts of equipment you have in your environment, I’m not so sure I can provide any real help.
-
Hi,
Actually, I have an IBM Bladecenter H chassis with 2 Nexus 4001l switches attached to 2 Nexus 5048 and 3 MDS 9148.
I can send you the configs if needed.
I used UCS by mistake!
Thanks a lot!
-
Thanks Scott.
After I enable the Cos/Qos featre. the SW pick the WWN of the server. Thanks. -
Do the FC ports (like the 8 dedicated FC only ports on the expansion module for a 5548P) require an external Fabric?
I am just wondering if in a very dense 3-4 rack buildout if I could have FCoE enabled Servers talking to the 5548 and then hang my 3Par storage right off the FC dedicated ports on the 5548s as well?
Do I NEED an external traditional FC Fabric if I am using Nexus?
-Casper42
-
Casper42,
You can indeed use the FC ports on a Nexus 5548 for connectivity to a storage array. The Nexus 5000 series switches have fabric services functionality available (name server, etc.) so that you don’t have to have an external Fibre Channel switch if you don’t need the additional port density.
Hope this helps!
-
We are diving into fcoe with our 2 5010′s and inbetween our 5010′s is a Cisco 6509 (sup720, both sides) with full layer 2 connectivity between our sites. If I setup a fcoe interface w/ vlan for fcoe, can I span that vlan across our 6509s (at layer 2) and still communicate with the other fcoe interface w/vlan on the other side? Thanks.
-
@Rob – I think you are asking if you can tunnel FCoE traffic across a Catalyst Switch. To my knowledge there is no FCoE support in the Catalyst family so the answer would be no.
FCoE requires DCB (aka CEE and formerly called DCE by Cisco) in order to function, especially in a Multi Hop environment. I don’t think Catalyst supports that feature which would also make me lean towards No again.
But perhaps someone smarter than I on Cisco gear can say for sure.
-
Great Post! I will be setting up two Nexus 5020′s very shortly. This will be new to me so I had a question about the distributed fabric. I will be connecting the two 5020′s to two existing MDS switches. Once I’ve created either the EISL or san port channel between the nexus and the mds is there anything else that needs to be done for the MDS to see the pwwwn’s of anything connected to the Nexus, I guess what I want to know is after the port channel is setup, when I type sh flogi database on the MDS will it also show devices that are only connected to the nexus 5020?
-
Just for the record, setting the switch to NPV DOES wipe out the entire configuration…..learning the hard way
-
Thanks for this post! SAN fabric management is new to me and I’m attempting to help deploy a pair of Nexus 5000′s with NetApps. This has helped me get a better start!
Because FCoE is new to me, I did a lot of googling. Thought you should know that this post was lifted by another blogger at http://www.druid.co.il/wordpress/?p=133
-
when y put
vlan 1
fcoe vsan 10the conectivity between my ucs blade servers and the nexus hangs also the nexus 5000 stops responding.
anyone??? some help??
thanks in advance
-
Isaac, we’re going to need a bit more detail than just a couple of commands taken out of context. Can you shed a bit more light on the problem and what you’re trying to accomplish?
-
Ok… sorry.. ill explain it better…:
I am trying to connect some cisco blades to a netapp storage trough a nexus 5000.
I have 2 cisco UCS fabric 6120 connected via fcoe to a nexus 5000 in ports:
Ethernet 1/1 = Fabric UCS 6120 1
Ethernet 1/2 = Fabric UCS 6120 2And to the ucs 6120 i have connected a cisco blade system with 2 b200
A fiber cable connected to a expansion module on the nexus 5000 to port and on the other end a netapp:
FC 2/2 = Netapp
………………………
Im following this guide doing this
vsan database
vsan 10vlan 2
fcoe vsan 10
no shutdowninterface vfc4
bind interface Ethernet 1/2
switchport trunk allowed vsan 10
no shutdownvsan database
vsan 10 interface vfc4
…………………………………………………the problem is that as soon as i put the command
vlan 2
fcoe vsan 10i loose conectivity to the blade servers and the conection to the management port on the nexus 5000 starts flapping…. i have enabled the features fcoe and npiv…. Im not talking about zonning cause i cant even get to that point cause i lose conection.
thanks
-
Isaac, I’m traveling at the moment and I’d really need to sit down and take a closer, more focused look at what you’re trying to accomplish. However, at first glance, try not setting vfc4 to a trunk with a specific allowed VSAN. Also, you didn’t mention what VLAN your management port is using, and you didn’t mention how connectivity to the management port is managed. Any additional information you can provide might be useful.
-
Hi…
I just figured out my problem…. its just not supported to get the ucs 6120 (blades) vhba through the fcoe on the nexus 5000…. the only way to do it is trough the fc uplinks on the expansion fc module. The nexus 5000 is not the problem, it supports it if fc is traveling through the fcoe link but in the case of ucs 6120 the only way to send fc data from the ucs 6120 to the nexus 5000 is through the fc expansion module of the ucs 6120.
Thanks
-
VLAN 1 will be used for negotiation, I would strongly suggest to use some other VLAN to map VSAN and on trunk allow VSAN Vlan, data vlans and VLAN 1.
-
Milena, thanks for the recommendation. I also generally recommend staying away from the use of VLAN 1 and VSAN 1 and using other VLANs and VSANs in your configurations.
-
Hi Scott,
I did the configuration between Nexus 5548UP and C-200 with P81E VIC. But i discovered that when the VIC finds and brings up the OS from my FC Storage after the loading of Operating System the flogi database is disappears from N5548UP. And also my C200 can use the ESXi OS but cannot see the datastores. There is no such a problem like this with my FIs and Blades. But i think i’m missing something on the N5548UP or on the VIC.
Do you have any advises for me?
Thank you.
-
Hi Scott,
I think on the Nexus 5548 and 5596 Plattform you also need to adjust QoS for FCoE. Because with the enable the feature fcoe, the Nexus will create the new service-policy, but not activate them:n5548(config)# system qos
n5548(config)# service-policy type qos input fcoe-default-in-policy
n5548(config)# service-policy type network-qos fcoe-default-nq-policy
n5548(config)# service-policy type queuing input fcoe-default-in-policy
n5548(config)# service-policy type queuing output fcoe-default-out-policyIs that correct?
Thanks a lot -
Thanks for very informative article. It is simple to understand.
-
Hi…
i have a problem about the nexus 5548up connected to the 6506.
i am follwing this guide
The nexus 5548up:
vsan datebase
vsan 10
vlan 10
fcoe vsan 10
interface vfc1
bind interface Ethernet 1/1
no shutdown
vsan datebase
vsan 10 interface vfc 1
interface ethernet 1/1(this port is connected to the HP980 sevrer which is supported the FCOE)
switchport mode trunk
no shutdown
interface ethernet 1/15(this port is connected to the cisco 6506,and this port type is RJ-45)
switchport mode trunk
speed is 1000
no shutdown
The cisco 6506:
vlan 10
interface vlan 10
ip address 192.168.100.254 255.255.255.0
interface g 1/0/1(this port is connected to the nexus 5548up)
switchport trunk en dot1q
swichport mode trunk
speed 1000
the HP980 address is 192.168.100.210 255.255.255.0
the problem is that the HP980 ping 192.168.100.254 unsuccessfull
why?thank you -
I’ve a customer who bought a Blade System with Flex Fabric, and a Cisco Nexus 5000. I was under the impression that FF could connect directly into this. However the information I have since received is that there has to be an additional Cisco layer of Fabric Extender switches between the two, with a Cisco 2000 series Fabric Extender switch added to the blade chassis.
I can see their value in effectively being repeaters, with the intelligence remaining in the Core switch.
However if physical port connectivity is available I don’t understand why we cannot connect directly to the Cisco 5000?
-
Mike, You can definately connect Flex Fabric ( assuming you’re talking about the HP branded 10gb blade switches?) directly to the Nexus 5k.
If you’re using FCoE in the Flex Fabric you’ll need to convert some of your ports to FC on the 5k for FC uplinks from the Flex Fabric Switches, but there are no real caveats to connecting the FlexFabric switches networking straight to the 5k.
-
Full Disclosure, I work for HP:
FlexFabric Ethernet direct to Nexus 5000 works fine.
I personally setup a PoC environment around Sept last year with exactly this design and there were no problems. Several other peers of mine have done similar as well.Now FlexFabric FC ports however into a Nexus 5000 FC are currently unsupported. There are interoperability issues between the QLogic chipset inside FlexFabric and the Nexus FC ports. I know Qlogic is working on it but I haven’t checked on the status in a while.
-
Mikemcg
Just about anything you do with a Fabric Extender you can do locally on a 5k. Are you using Multiple 5k’s with multiple Flex Fabric?If I had to guess, you are trying to create a VPC between the two Flex Fabric and 2 5k’s. We are having some similar issues to what you are seeing. I only work on the network side, but the Blade deployment is very different here.
If you use a Fex 2232 then that would give you a bit more redundancey however that comes with a draw back. It will also give you 1 point of failure if that 2232 crashes (assuming you are using 1 2232).
We will have an HP consultant come in here in the next week to help the Server team setup the blade chassis.
Site Sponsors
38 comments
Comments feed for this article
Trackback link: http://blog.scottlowe.org/2009/10/25/setting-up-fcoe-on-a-nexus-5000/trackback/