Setting up Software FCoE on vSphere 5

One of the features added in vSphere 5 was a software FCoE initiator. This is a feature that has gotten relatively little attention, other than a brief mention here and there. I’m not entirely sure why the software FCoE initiator in vSphere 5 hasn’t gotten more attention, but this past week I had the opportunity to work with the software FCoE initiator in a bit more detail. In this post I’m going to describe in a bit more detail how to set up the software FCoE initiator; in future posts, I hope to be able to provide some performance and troubleshooting information.

Prerequisites

In order to use the software FCoE initiator, you must have a network interface card (NIC) that supports the FCoE offloads. The Intel X520 NICs certainly do; these are the cards that I used in my testing. (Disclaimer: Intel provided me a pair of X520 NICs at no charge to use in this testing.) There might be others, I don’t know; the VMware HCL doesn’t seem to provide an easy way of identifying those NICs that do support the FCoE offloads vs. those NICs that don’t.

If you have a NIC that doesn’t support FCoE offloads, you won’t even be able to add a software FCoE initiator:

If, on the other hand, your NIC does support FCoE offloads, you’ll see the option to add a software FCoE initiator, like this:

Obviously, you’ll want to be sure that your NIC does support FCoE offloads before continuing.

Setting Up Networking for Software FCoE

Before you can actually set up a software FCoE initiator, you’ll first need to configure your networking appropriately. There is one important piece of information you’ll want to be sure to have before you continue: the ID of the VLAN for FCoE traffic.

If you’ve read my article on setting up FCoE on a Nexus 5000, then you know that—on a Nexus 5000 series switch, anyway—you must map the FCoE VSAN to a VLAN (using the fcoe vsan XXX command). You’re going to need that VLAN ID, so make sure that you have it. In a dual fabric setup, you’re going to have two different VLAN IDs. You’ll need both.

Once you have those VLAN IDs, you can then proceed with the networking setup:

  1. Create a VMkernel port on your ESXi host. When creating the VMkernel port, when prompted for VLAN ID specify the FCoE VLAN on fabric A.
  2. I don’t know why (and maybe this will be fixed in a future release), but you’ll need to assign an IP address to each VMkernel port. The IP address will not be used, so just pick an address on a subnet that you don’t use. (Don’t use a subnet that you are using elsewhere on the ESXi host or you could run into IP routing weirdness—remember that the VMkernel uses the first interface it finds for a particular route.)
  3. After you’ve created the VMkernel port, modify the NIC teaming policy to only use the physical uplink that is physically connected to fabric A. This might require a bit of sleuthing and/or using CDP/LLDP data to ensure that you have the right uplink selected.
  4. Create a second VMkernel port on your ESXi host, this time specifying the FCoE VLAN ID for fabric B and modifying the NIC teaming policy
    When creating the VMkernel ports, specify the appropriate VLAN IDs—one for fabric A, one for fabric B. Modify the NIC teaming policy to only use the physical uplink connected to fabric B, again using physical tracing and CDP/LLDP data as needed to verify it.

At this point, you should now have two VMkernel ports, each with separate (unused) IP addresses and configured for separate VLAN IDs and separate physical uplinks. The VLAN IDs and the physical uplinks should correspond to the FCoE fabrics (fabric A and fabric B).

Setting Up the Software FCoE Initiator

With the networking in place, you can actually add the software FCoE initiator using the “Add…” button on the Storage Adapters view of the Configuration tab. This opens up the Add Software Adapter dialog box I showed you earlier, where you can select “Add Software FCoE Adapter” and click OK.

That option opens the following dialog box:

You’ll note that the VLAN ID is 0 and isn’t changeable. I couldn’t find any way to enable this field, and in my testing it proved unnecessary to change it (it worked anyway). Select the appropriate physical uplink and click OK. You’ll do this twice—once for fabric A, and again for fabric B. After you’ve done this twice, you’ll have two software FCoE adapters:

For each of these two software FCoE adapters, you’ll see a node WWN and a port WWN listed. You can use these values in creating the zones and zonesets (see here for more information). First, though, you’ll want to be sure that the software FCoE adapter is actually talking to the fabric correctly; the best way to do that is to use the show flogi data command (on a Nexus 5000; other vendors’ switches will use a slightly different command). The outcome of the show flogi data command will look something like this:

In this screenshot (taken from an SSH session into a Nexus 5010), you can see that a device has logged into vfc1009 on VSAN 300. If you compare the port name and node name, you’ll see that they match one of the software FCoE adapters shown earlier. This is only one of the two fabrics; a matching result was seen from the other fabric, showing that both software FCoE adapters successfully logged into their respective fabrics.

At this point, the rest of the configuration consists of creating the appropriate device aliases (if desired); defining zones and zonesets; and presenting storage from the storage array(s) to the initiator(s). Since these are topics that are fairly well understood and well documented elsewhere, I won’t rehash them again here.

In future posts, I hope to be able to do provide some performance and some troubleshooting information. However, it will likely be early 2012 before I have that opportunity. In the meantime, if anyone has any additional information they’d like to share—including any corrections or clarifications—please feel free to add them to the comments below.

Tags: , , , ,

11 comments

  1. Donny’s avatar

    Scott,

    Thank you for presenting another facet of vSphere 5. However, I believe that FCoE is dying on the vine. Outside of Cisco platforms, there is little if any demanad. iSCSI, on the other hand, is rampant.

    If you are moving to an ‘Ethernet’ fabric, there is little need to bring along FC. iSCSI or NFS is a more viable solution in these situations than FCoE. If an environment has a high investment in FC, the upgrade will most likely utilize an FC fabric to drive utilization of investment.

    So, while it is good to have the flexibility within VMware to create FCoE connectivity natively, the market for FCoE products and technologies is a single horse race.

  2. slowe’s avatar

    Donny, there are plenty who would disagree with you regarding the fate/state of FCoE, but that’s another discussion for different day! Thanks for taking the time to read and comment.

  3. Cormac’s avatar

    Hey Scott,

    Nice post. Something I’ve been planning to do for some time but just haven’t got around to it yet.

    In step 1 & step 4, there is no need to specify a VLAN ID for the VMkernel port. The VLAN ID will be automatically discovered during the FIP VLAN discovery process. This is also the reason why you cannot edit this field in the ‘Add Software FCoE Adapter’ dialog box.

  4. slowe’s avatar

    Cormac, thanks for that tidbit—now that you point that out, it makes sense.

  5. iwan 'e1' rahabok’s avatar

    Hi Scott, thanks for sharing.
    The CNA needs to split between bandwidth vmhba and vmnic. Does the CNA hardcode the limit for each, or is it smart enough to do sharing? Example, there is little network traffic (1 GE) but heavy storage traffic (9 GE), can the vmhba utilise 9 GE, or it will be capped to 4 GE (say this is what is configured). If it is hardcoded, it would be waste.

    Another question, how does LBT work in this case? Say we split the CNA into 4 GE for vmhba and 6 GE for vmnic. Does LBT treat 6 as the physical limit? We know that LBT is based utilization of an uplink. But I’m not 100% certain how utilization is determined (vCenter Ops use a different logic to determine utilisation and congestion, so I won’t assume I understand the vSphere logic).

    Thanks again Scott.

  6. iwan 'e1' rahabok’s avatar

    Hi Scott,

    On the sentence “I don’t know why (and maybe this will be fixed in a future release), but you’ll need to assign an IP address to each VMkernel port”, my guess is it uses the same dialog box, and that dialog box requires the IP Address to be filled.

    The reason why I came up with that guess is the Distributed Switch dialog box has the same behaviour. It uses the same dialog box for uplink and port-group, so it’s a bit awkward for the port group. I was trying to understand why information is shown but then always grey out, when I realised that the info was not relevant (but can’t be hidden as it’s the same dialog box).

    Cheers!
    e1

  7. Vamsi’s avatar

    Hi Scott,

    What is the need of having an Intel X520 NICs for the “FCOE Sowftware initiator” to work?
    The iSCSI sowftware initiator never had the requirement for any specific nic installed in the server.
    Thank You
    Vamsi

    By the way “Master vsphere 5″ book is great manual.

  8. slowe’s avatar

    Vamsi, thank you for the feedback on the book. Regarding software FCoE, the software initiator was simply built to require certain functionality out of the NICs. I can’t attest as to the factors that led VMware to make this particular design decision, but they did. (Perhaps there is a reader from VMware that can enlighten us regarding this decision?) The fact that the software iSCSI initiator doesn’t have these requirements is a separate matter and isn’t, in my opinion, related. Thanks for your comment!

  9. Pierre-Louis’s avatar

    Hi Scott,

    I am busy working an use case with software FCoE and Nexus 5k and I thought that you would be the good person to help on this…

    I can’t manage to get a flogi on my fabric, only one port connected. The setup is approximately the same you described, but with Broadcom 10G CNA adapter (BCM57810S, with latest VMware drievrs) on a Dell R710 hardware server, with VMware ESXi 5.0 installed. I’ve tried to enable VLAN on Net Adapter, delete it on VMware, many techniques, but nothing is working for me.

    Previously we had an Emulex CNA with hardware FCoE talking to NetApp Disk Array on our Fabric, no problem. What a good idea to change our CNA adapters !…

    I have logs, Printscreens and other details if needed.

    Thanks a lot for your help

    /plg

  10. Matt’s avatar

    What’s the difference between a x520/x540 software FCoE CNA vs an Emulex hardware one besides one is 10GBaseT RJ45 vs SFP+?
    Is it that the hardware ones still allow for the NIC and HBA to run off the same card but that the HBA is automatically discovered and avail w/o needing to add the Software FCoE Initiator?
    Or, do ALL FCoE CNAs still require you to add a Software FCoE Initiator in ESX?
    Thanks!

  11. jasonsung’s avatar

    Is there some problems?multipath problems? with intel x520 fcoe initiator?

Comments are now closed.