Before I can describe NIV, I need to define a few key terms:

IV-capable bridge: This is a switch that understands and is aware of interface virtualization (IV). Examples of IV-capable bridges include the Nexus 5000 and the UCS 6100XP fabric interconnects.

Interface virtualizer: An interface virtualizer (IV) is a device that simply extends the reach of an IV-capable bridge. To the outside world, an IV-capable bridge and all of its IVs appear as a single bridge. Examples of IVs include the Nexus 2000 fabric extender and the I/O module (IOM) in the Cisco Unified Computing System (UCS). There are also other forms of IVs, as I’ll explain later in this post.

Link-local tag: Frames entering an IV from a network interface card (NIC) have a link-local tag added to them; this link-local tag denotes the source IV port. Similarly, frames entering an IV-capable bridge have a link-local tag added to them that indicates the path through the IV(s) to the destination IV port. In this way, the link-local tag supplants the MAC address as the primary method of determining to which port (or out which port) a frame should be forwarded. The link-local tag is removed from the frame when it exits an IV (headed into a NIC) or when it exits an IV-capable bridge. (This link-local tag is what Cisco refers to as VNTag.)

Now that I’ve defined some NIV terminology, I’d like to explain why NIV is useful. To understand the value of NIV, take a look at the progression or development of data center networking:

1. Before virtualization, bridges (or switches) were connected to a NIC (sometimes multiple NICs, but usually just one NIC) in a physical server. The relationship between physical NICs, physical switch ports, and MAC addresses is static and easy to determine (generally one NIC with one MAC address connected to one switch port). The bridge hierarchy is reasonably simple.
2. Then hardware manufacturers introduce the server blade form factor. This introduces another layer of bridges (switches) in the blade chassis themselves and creates a more complex bridge hierarchy. This more complex bridge hierarchy also means more management, as each of these bridges represents a point of management. In order to provide redundancy between the various layers of bridges multiple connections are necessary; this, in turn, necessitates the use of Spanning Tree Protocol (STP) in order to prevent bridging loops. STP creates active/passive connections, meaning the effective bandwidth of the network is reduced.
3. Along comes virtualization, which introduces the idea of multiple virtual NICs (vNICs) associated with a single physical NIC. This, in turn, introduces another layer of switching and more points of management, and the bridge hierarchy grows more complex. Because multiple MAC addresses are now associated with a single NIC connected to a single switch port, bridges must now deal with “hairpin forwarding,” in which the switch must retransmit a frame out the same port on which it was received. The relationship between NICs, MAC addresses, and switch ports is now much more complex and very dynamic (due to live migration technologies).

As the proliferation of virtualization continues, this trend toward increased complexity also continues unabated. How, then, are we supposed to address this? NIV is intended to help address this problem. NIV seeks to remove the complexity from the edge—the NICs and vNICs—and drive that complexity toward the bridges. That is a key underlying principle behind NIV. Look back at the definitions: one characteristic of an IV-capable bridge is that the IV-capable bridge and all of its associated IVs appear to the outside world as a single bridge.

For example, consider a Nexus 2000 fabric extender connected to a Nexus 5000 switch. From both a management perspective as well as a networking perspective, the Nexus 5000+Nexus 2000 combination appears as a single device. The Nexus 2000 is simply an extension of the Nexus 5000 (hence the name “fabric extender”). This is why we say that a Nexus 2000 is an example of a interface virtualizer and why a Nexus 5000 is provided as an example of an IV-capable bridge. Similarly, this is why an IOM in the back of a UCS blade chassis functions as an interface virtualizer; it acts as an integrated part of the UCS 6100XP fabric interconnect. Just as the ports on a Nexus 2000 appear to be part of the Nexus 5000, the ports on a UCS IOM appear to be part of the UCS 6100XP fabric interconnects.

By now it should start making a little bit of sense. IVs allow you to scale to larger port densities, but they keep the complexity away from the edge of the network. If you’re astute, though, you’ll note that the examples I’ve provided so far don’t address the growth of vNICs created by virtualization. The Nexus 2000 and the UCS IOM help address the growth of physical ports, but not virtual ports. Can NIV help address vNICs as well?

So far, the examples I’ve provided of IVs have been IVs that embrace the bridge/switch form factor. There’s no reason, though, that an IV must look or feel like existing bridges or switches. Here’s another type of interface virtualizer that will really blow you away: what about the Cisco Virtual Interface Controller (VIC), aka Palo? Think about it: a VIC is really just an IV built into a UCS server blade. It appears to the outside world as additional ports on the IV-capable bridge (the UCS 6100XP fabric interconnect) to which it is connected. This underscores the flexibility of IVs and also underscores the fact that IVs can be “chained,” just as the VIC (an IV built onto the server blade) is “chained” behind the UCS IOM (an IV in the rear of the UCS chassis). Chained IVs appear as part of the upstream IV-capable bridge to which they are connected.

By building the IV into the server itself and leveraging hypervisor bypass (VMDirectPath in VMware vSphere), Cisco can address the growth of vNICs. With VIC as an IV on a VMware vSphere host, the ports that connect directly to a VM appear as ports on the upstream IV-capable bridge (the UCS 6100XP in this case); this erases any distinction between physical NICs on physical servers and virtual NICs on virtual servers. Again, a key component of NIV is that an IV-capable bridge and all associated IVs (regardless of form factor) appear as a single bridge.

Related to NIV is the idea of an Ethernet Host Virtualizer (EHV). EHV describes the behavior when an IV-capable bridge and associated IVs appear to the rest of the network as a single host. Because it appears as a single host, all issues with multiple uplinks and STP now go away. This is why, for example, the uplinks on the UCS 6100XP fabric interconnects are always active-active. I’m planning on delving a bit deeper in EHV in the near future.

I hope that this discussion of NIV has been useful. If you are interested in some additional information, I found some documents which were extremely helpful in solidifying my information. These documents (here, here, and here) are very technical but they are good sources of information nevertheless.

In a future post, I’ll discuss the role of the Nexus 1000V in NIV and how it relates to the other components described here.

Feel free to post any questions, comments, or clarifications below.

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

Understanding Network Interface Virtualization

• Understanding NPIV and NPV
• Connecting Nexus 5000 to Older Gigabit Ethernet Switches
• Identifying ESX Server NICs in Blades
• No Such Thing as an End-to-End FCoE Solution
• Why No Multi-Hop FCoE?

In addition, my blogging frequency was down this past year because of the two books I wrote this year, Mastering VMware vSphere 4 and VMware vSphere 4 Administration Instant Reference (with Jase McCarty and Matthew Johnson).

Still, the site has a sizable following. I’m averaging around 7,700 RSS subscribers, and the site is getting around 100,000 visits a month (depending upon which statistics mechanism you use, sometimes higher and sometimes lower). Anecdotally, I’ve had a few other bloggers tell me that their traffic spikes whenever I link to one of their posts. I guess I must be doing something right!

Luckily, I have gotten a few good posts out this year:

Creating a Bootable ESXi USB Stick on Mac OS X
vSphere Virtual Machine Upgrade Process
VMware vSphere vDS, VMkernel Ports, and Jumbo Frames
New User’s Guide to Configuring VMware ESX Networking via CLI
Introduction to Nehalem Memory (by guest author Aaron Delp)
Using VMware ESX Virtual Switch Tagging with HP Virtual Connect
Another Reason Not to Use PVSCSI or VMXNET3

And those are just the virtualization-related posts…I’ve also expanded into more SAN-related posts (a couple of Cisco MDS posts here and here), an introductory post on SR-IOV here, and some FCoE posts (here and here, for example).

So, if you could find it in your hearts to vote for me, I’d certainly appreciate it. Go vote now!

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

Vote Early (But Don’t Vote Often)

• VMware vSphere vDS, VMkernel Ports, and Jumbo Frames
• Enabling Jumbo Frames on a Nexus 5000
• VMware ESX Networking Articles
• ESX Server, IP Storage, and Jumbo Frames
• Vote for My Site!

• Scott Drummonds posted a good article with a performance comparison of thick provisioned disks vs. thin provisioned disks. This is good information and helps to clear up a lot of misinformation about the behavior of thin provisioned disks.
• Interested in using the software iSCSI initiator with a dvSwitch and multiple paths? This article can give you the information you need to get started.
• Frank Denneman has a great article providing some details on memory reservations and memory usage. It’s definitely worth a read, as is this post on the impact of guest OS type mismatch. See, there is a reason why it’s important to make sure you select the correct guest OS type when creating VMs.
• I’ve written about the paravirtualized SCSI (PVSCSI) driver before, but Scott Sauer over at Virtual Insanity has a great two-part series on PVSCSI that is definitely worth a read (part 1 and part 2).
• Although a bit dated now, Dave Lawrence has a great review of some of November’s technical white papers. He references the thin provisioning performance paper that’s also referenced in Scott Drummond’s post above, as well as a white paper on optimizing SRM performance. Thanks for bringing our attention to these documents, Dave!
• The release of VMware View 4 brings with it PCoIP, which is supposed to bring enhanced performance over older display protocols. Unfortunately, everything I was hearing was that PCoIP was incompatible with many WAN acceleration solutions. So I was quite puzzled at this press release. Reading the press release a bit more closely, though, it would seem that Expand’s solution doesn’t actually optimize or accelerate PCoIP; rather, it enables it to be tunneled and applies Quality of Service (QoS). Something is better than nothing, I guess.
• Rick Scherer describes a strange issue with vCenter Server 4.0 Update 1 and VMware ESXi 4.0 Update 1. Rick was seeing a number of strange symptoms, including ESXi hosts suddenly disconnecting from vCenter Server. Last time I checked Rick still hadn’t identified the root cause, although the symptoms he was seeing have since disappeared.
• Stu provides a thorough explanation of why VMware is recommending not to install vCenter Server 4.0 Update 1 when managing VMware ESXi 4.0 hosts.
• Need to install the HP Management Agents on VMware ESX 4.0? Here are some helpful instructions.
• Nigel Poulton has started a great series on rack area networks (RANs), of which a key component is I/O virtualization. So far, Nigel has published part 1 (introducing the concept of a RAN), part 2 (IOV’s role in a RAN), and part 3 (IOV and hairpin turns). Part 2, in particular, has a good discussion of SR-IOV and MR-IOV. Nigel’s discussion of SR-IOV is a good complement to my own.
• Greg Schulz also has a lengthy article on I/O virtualization.
• Curious to know why NUMA is important with vSphere? Network Computing blogger Jake McTigue (with whom I had the honor of participating on a recent virtual networking webcast) has a good overview of NUMA and what it means for virtualized environments. It’s worth a read if you’re not already familiar with NUMA.
• Need more information on storage alignment and VMFS block sizes? Check out this VIOPS document.

I also have a whole list of other links that I haven’t had the chance to read yet but that look like they might be interesting or useful:

A handy new addition to the Command Line Tool for View 4
Whats what in VMware View and VDI Land
How to get PCoIP with View 4 to work every time!
Revisiting the Components of the Cisco Nexus 1000v
File Virtualization… The short primer
Is Your Blade Ready for Virtualization? A Math Lesson
RSA SecureBook for VMware View hardening now publicly available!

I guess that’s enough for now. If you have any other useful, unique, or interesting virtualization-related links, feel free to share them in the comments. Thanks for reading!

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

Virtualization Short Take #32

• Virtualization Short Take #35
• Virtualization Short Take #30
• Virtualization Short Take #33
• Virtualization Short Take #27
• Virtualization Short Take #36

Virtual Multi-port Software allows for consolidation of switch ports and cabling by using 10Gb infrastructure while maintaining the existing Gigabit-based ESX setup. The software enables the consolidation by keeping the infrastructure update completely transparent to the ESX hypervisor, enabling a 10Gb adapter to appear to the hypervisor as eight virtual Gigabit adapters. By offloading the tasks performed by the hypervisor, the Chelsio adapters can deliver the best I/O performance for virtualized applications.

There’s no mention of SR-IOV (more information on SR-IOV is available in this post), so I’m guessing that this is a proprietary technology similar to what HP is using in Virtual Connect Flex-10. The key difference with HP Virtual Connect Flex-10 and the Chelsio solution is that Flex-10 doesn’t require any software support in the OS or hypervisor, whereas Chelsio’s solution does require software support (as does SR-IOV). Nevertheless, it’s clear that I/O virtualization—even relatively simple forms of I/O virtualization such as this—is gaining more and more attention.

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

Multiport Uplink Driver

• Cisco UCS Virtualization-Optimized CNAs
• Thinking Out Loud: HP Flex-10 Design Considerations
• Follow-Up About Multiple VLANs, Virtual Connect, and Flex-10
• Using VMware ESX Virtual Switch Tagging with HP Virtual Connect
• Thinking Out Loud: It’s Not Just the Technology

First, though, I need to cover some basics. This is unnecessary for those of you that are Fibre Channel experts, but for the rest of the world it might be useful:

• N_Port: An N_Port is an end node port on the Fibre Channel fabric. This could be an HBA (Host Bus Adapter) in a server or a target port on a storage array.
• F_Port: An F_Port is a port on a Fibre Channel switch that is connected to an N_Port. So, the port into which a server’s HBA or a storage array’s target port is connected is an F_Port.
• E_Port: An E_Port is a port on a Fibre Channel switch that is connected to another Fibre Channel switch. The connection between two E_Ports forms an Inter-Switch Link (ISL).

There are other types of ports as well—NL_Port, FL_Port, G_Port, TE_Port—but for the purposes of this discussion these three will get us started. With these definitions in mind, I’ll start by discussing N_Port ID Virtualization (NPIV).

N_Port ID Virtualization (NPIV)

Normally, an N_Port would have a single N_Port_ID associated with it; this N_Port_ID is a 24-bit address assigned by the Fibre Channel switch during the FLOGI process. The N_Port_ID is not the same as the World Wide Port Name (WWPN), although there is typically a one-to-one relationship between WWPN and N_Port_ID. Thus, for any given physical N_Port, there would be exactly one WWPN and one N_Port_ID associated with it.

What NPIV does is allow a single physical N_Port to have multiple WWPNs, and therefore multiple N_Port_IDs, associated with it. After the normal FLOGI process, an NPIV-enabled physical N_Port can subsequently issue additional commands to register more WWPNs and receive more N_Port_IDs (one for each WWPN). The Fibre Channel switch must also support NPIV, as the F_Port on the other end of the link would “see” multiple WWPNs and multiple N_Port_IDs coming from the host and must know how to handle this behavior.

Once all the applicable WWPNs have been registered, each of these WWPNs can be used for SAN zoning or LUN presentation. There is no distinction between the physical WWPN and the virtual WWPNs; they all behave in exactly the same fashion and you can use them in exactly the same ways.

So why might this functionality be useful? Consider a virtualized environment, where you would like to be able to present a LUN via Fibre Channel to a specific virtual machine only:

• Without NPIV, it’s not possible because the N_Port on the physical host would have only a single WWPN (and N_Port_ID). Any LUNs would have to be zoned and presented to this single WWPN. Because all VMs would be sharing the same WWPN on the one single physical N_Port, any LUNs zoned to this WWPN would be visible to all VMs on that host because all VMs are using the same physical N_Port, same WWPN, and same N_Port_ID.
• With NPIV, the physical N_Port can register additional WWPNs (and N_Port_IDs). Each VM can have its own WWPN. When you build SAN zones and present LUNs using the VM-specific WWPN, then the LUNs will only be visible to that VM and not to any other VMs.

Virtualization is not the only use case for NPIV, although it is certainly one of the easiest to understand.

<aside>As an aside, it’s interesting to me that VMotion works and is supported with NPIV as long as the RDMs and all associated VMDKs are in the same datastore. Looking at how the physical N_Port has the additional WWPNs and N_Port_IDs associated with it, you’d think that VMotion wouldn’t work. I wonder: does the HBA on the destination ESX/ESXi host have to “re-register” the WWPNs and N_Port_IDs on that physical N_Port as part of the VMotion process?</aside>

Now that I’ve discussed NPIV, I’d like to turn the discussion to N_Port Virtualization (NPV).

N_Port Virtualization

While NPIV is primarily a host-based solution, NPV is primarily a switch-based technology. It is designed to reduce switch management and overhead in larger SAN deployments. Consider that every Fibre Channel switch in a fabric needs a different domain ID, and that the total number of domain IDs in a fabric is limited. In some cases, this limit can be fairly low depending upon the devices attached to the fabric. The problem, though, is that you often need to add Fibre Channel switches in order to scale the size of your fabric. There is therefore an inherent conflict between trying to reduce the overall number of switches in order to keep the domain ID count low while also needing to add switches in order to have a sufficiently high port count. NPV is intended to help address this problem.

NPV introduces a new type of Fibre Channel port, the NP_Port. The NP_Port connects to an F_Port and acts as a proxy for other N_Ports on the NPV-enabled switch. Essentially, the NP_Port “looks” like an NPIV-enabled host to the F_Port on the other end. An NPV-enabled switch will register additional WWPNs (and receive additional N_Port_IDs) via NPIV on behalf of the N_Ports connected to it. The physical N_Ports don’t have any knowledge this is occurring and don’t need any support for it; it’s all handled by the NPV-enabled switch.

Obviously, this means that the upstream Fibre Channel switch must support NPIV, since the NP_Port “looks” and “acts” like an NPIV-enabled host to the upstream F_Port. Additionally, because the NPV-enabled switch now looks like an end host, it no longer needs a domain ID to participate in the Fibre Channel fabric. Using NPV, you can add switches and ports to your fabric without adding domain IDs.

So why is this functionality useful? There is the immediate benefit of being able to scale your Fibre Channel fabric without having to add domain IDs, yes, but in what sorts of environments might this be particularly useful? Consider a blade server environment, like an HP c7000 chassis, where there are Fibre Channel switches in the back of the chassis. By using NPV on these switches, you can add them to your fabric without having to assign a domain ID to each and every one of them.

Here’s another example. Consider an environment where you are mixing different types of Fibre Channel switches and are concerned about interoperability. As long as there is NPIV support, you can enable NPV on one set of switches. The NPV-enabled switches will then act like NPIV-enabled hosts, and you won’t have to worry about connecting E_Ports and creating ISLs between different brands of Fibre Channel switches.

I hope you’ve found this explanation of NPIV and NPV helpful and accurate. In the future, I hope to follow up with some additional posts—including diagrams—that show how these can be used in action. Until then, feel free to post any questions, thoughts, or corrections in the comments below. Your feedback is always welcome!

Disclosure: Some industry contacts at Cisco Systems provided me with information regarding NPV and its operation and behavior, but this post is neither sponsored nor endorsed by anyone.

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

Understanding NPIV and NPV

• VIR250: Advanced Storage Connectivity for VMs
• Understanding Network Interface Virtualization
• New User’s Guide to Configuring Cisco MDS Zones via CLI
• Why No Multi-Hop FCoE?
• Setting Up FCoE on a Nexus 5000

Connecting Nexus 5000 to Older Gigabit Ethernet Switches
Setting Up FCoE on a Nexus 5000
FCoE and VLAN Trunking on Nexus 5000

One thing I hadn’t yet documented was how to enable jumbo frames on a Nexus 5000. Since jumbo frames are now officially supported for VMkernel traffic with VMware vSphere, the combination of jumbo frames and 10Gb Ethernet is an attractive one. I’ve covered the ESX/ESXi side (ordinary vSwitches here and distributed vSwitches here), but here’s the Nexus side.

The commands are pretty straightforward, and I’ve included the commands for both NX-OS 4.0 and NX-OS 4.1 (they are different between versions). Important note: if you enabled jumbo frames under NX-OS 4.0 and then upgraded the switch to version 4.1, you’ll need to re-do your jumbo frame configuration.

For NX-OS 4.1, the commands to enable jumbo frames are:

switch(config)# policy-map type network-qos jumbo
switch(config-pmap-nq)# class type network-qos class-default
switch(config-pmap-c-nq)# mtu 9216
switch(config-pmap-c-nq)# exit
switch(config-pmap-nq)# exit
switch(config)# system qos
switch(config-sys-qos)# service-policy type network-qos jumbo

Now, contrast the commands above with the following commands, which you would have used to enable jumbo frames on NX-OS 4.0:

switch(config)# policy-map jumbo
switch(config-pmap)# class class-default
switch(config-pmap-c)# mtu 9216
switch(config-pmap-c)# exit
switch(config)# system qos
switch(config-system)# service-policy jumbo

The end result of these differences is this: if you upgrade NX-OS from 4.0 to 4.1, then your jumbo frames configuration will go away, and you’ll need to enter the commands for version 4.1 in order to enable jumbo frame support again. This little gotcha caused me quite a headache when my NFS-based datastores suddenly went offline after the NX-OS upgrade.

More information on the necessary commands can be found here for version 4.0 and here for version 4.1.

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

Enabling Jumbo Frames on a Nexus 5000

• Setting Up FCoE on a Nexus 5000
• New User’s Guide to Configuring Cisco MDS Zones via CLI
• New User’s Guide to Managing Cisco MDS Zones via CLI
• LACP with Cisco Switches and NetApp VIFs
• ESX Server, NIC Teaming, and VLAN Trunking

Acadia is interesting, but it really isn’t the meat of the announcement, in my opinion. The real substance of the matter is the nature of the coalition. There are many interesting questions/thoughts circling in my head right at the moment:

• What impact will this have on VMware’s relationship(s) with HP, IBM, and Dell? “Throwing their hat in the ring” with Cisco’s UCS, so to speak, may greatly endanger VMware’s much larger (with respect to revenue) relationships with other OEMs. What will happen to VMware if those OEMs “throw their hat in the ring” with Microsoft and Hyper-V? This is not a good place to be.
• The acrimonious Cisco-HP relationship adds further fuel to the concerns over VMware’s close alliance with Cisco’s computing platform.
• Does this new coalition signal a move away from the “arms-length” relationship between EMC and VMware, a move that some (competitors, notably) have been talking about for some time? If so, what danger does that put VMware in with regards to storage relationships?
• It seems to me that VMware has the most to lose here. What does EMC lose if this doesn’t go well? Nothing, really. What about Cisco? Nothing, really. VMware, on the other hand…well, it could be ugly.
• What does this coalition offer that the three companies couldn’t deliver without the coalition? Why risk important relationships? This is a big question in my mind. Lots of technology companies have delivered validated designs without any sort of formal coalition. Why is one necessary in this case?
• On the other end of the spectrum—keeping Acadia out of the picture for the moment—is this “new coalition” really anything more than what the three companies have already been doing? Is this really anything more than each of the companies dedicating resources to this effort? I know from my own direct interaction with at least one of these vendors that resources had already been dedicated to the VCE technology intersection before any sort of formal announcement. So, does this formal announcement really mean anything at all?

I don’t have any answers (yet), but you can at least read my thoughts—and contribute back to them via the comments—without having to pay $499 to some analyst firm.

By the way, if you’d like some other viewpoints on this matter, here are a couple from opposing viewpoints:

NetApp - Jay’s Blog: The Importance of Being Open
Chuck’s Blog: Announcing the VCE Coalition

Feel free to speak up in the comments below (courteous comments only, please, and be sure to include full vendor disclosure where appropriate). Thanks!

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

A Few Quick Thoughts on the VCE Coalition Announcement

• A Collection of Viewpoints on Cisco UCS
• Citrix to buy XenSource
• A Comment Policy Reminder
• Greene is Out; What’s Next?
• Cisco Switches on VMware

Fast forward to yesterday, when I was troubleshooting a problem between a Gen2 QLogic CNA and the Nexus 5010 in my lab (I tweeted about it). Although the Ethernet side of the CNA works just fine, the CNA refuses to bring up an FCoE connection. In the process of troubleshooting, Brad Hedlund (check his outstanding web site) suggested to me in a Twitter direct message that I should double-check the VLAN trunking status of the interface. That part I’d already heard from the reader who commented on the first post, but the next part was new to me (emphasis mine):

Gen2 requires ’switchport mode trunk’ on the 5K. Gen1 doesn’t. Also make sure FCoE VLANs are allowed on the trunk.

Ah, now there’s something I hadn’t heard! That prompted me to do a bit of testing this morning (yes, I know I’m supposed to be studying for the VCDX Design Exam this afternoon). In my testing, I confirmed that a Gen1 CNA (I’m using Gen1 Emulex CNAs) does not require VLAN trunking to be enabled on the Ethernet interface.

There does appear to be a “gotcha” though: if the Ethernet interface is in access mode, it’s access VLAN must be the same as the FCoE VLAN; otherwise, the vfc interface will report down.

In summary:

• If you are using a Gen2 CNA, you must put the Ethernet interface in VLAN trunk mode.
• If you are using a Gen1 CNA, the Ethernet interface may be in either access mode or trunk mode.
• If the interface is in trunk mode, be sure that you have allowed the FCoE VLAN via the switchport trunk allowed vlan command.
• If the interface is in access mode, be sure that you have placed the interface in the FCoE VLAN via the switchport access vlan command.

If there are any other subtleties or nuances I’ve missed, please post them in the comments below so that future readers will benefit. Thank you!

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

FCoE and VLAN Trunking on Nexus 5000

• Connecting Nexus 5000 to Older Gigabit Ethernet Switches
• Configuration for Protecting VMotion
• Setting Up FCoE on a Nexus 5000
• ESX Server, NIC Teaming, and VLAN Trunking
• Link State Tracking in Blade Deployments

1. Enable FCoE on the switch.
2. Map a VSAN for FCoE traffic onto a VLAN.
3. Create virtual Fibre Channel interfaces to carry the FCoE traffic.

The first step is incredibly easy. To enable FCoE on the switch, just use this command:

switch(config)# feature fcoe

The next part of the FCoE configuration is mapping a VSAN to a VLAN. What VSAN should you use? Well, if you are connecting to an existing Fibre Channel fabric, perhaps on a Cisco MDS switch, you’ll need to make sure that the VSANs between the Nexus and the MDS are appropriately matched. Otherwise, traffic on one VSAN on the Nexus won’t be able to reach devices on another VSAN on the MDS. If there’s enough demand, I’ll post a quick piece on this step as well.

Note that this FCoE VSAN-to-VLAN mapping is a required step; if you don’t do this, the FCoE side of the interfaces won’t come up (as you’ll see later in this post). Assuming the VSAN is already defined, perform these steps to map the VSAN to a VLAN:

switch(config)# vlan XXX
switch(config-vlan)# fcoe vsan YYY
switch(config-vlan)# exit

Obviously, you’ll want to substitute XXX and YYY for the correct VLAN and VSAN numbers, respectively.

After you’ve enabled FCoE and mapped FCoE VSANs onto VLANs, then you are ready to create virtual Fibre Channel (vfc) interfaces. Each physical Nexus port that will carry FCoE traffic must have a corresponding vfc interface. Generally, you will want to create the vfc interface with the same number as the physical interface, although as far as I know you are not required to do so. It just makes management of the interfaces easier. The commands to create a vfc interface look like this:

switch(config)# interface vfc ZZ
switch(config-if)# bind interface ethernet 1/ZZ
switch(config-if)# no shutdown
switch(config-if)# exit

At this point the vfc interface is created, but it won’t work yet; you’ll need to place it into an VSAN that is mapped to an FCoE enabled VLAN. If you don’t, the show interface vfc <number> command will report this (emphasis mine):

vfc13 is down (VSAN not mapped to an FCoE enabled VLAN)

As I mentioned earlier, if you haven’t mapped the FCoE VSAN onto a VLAN, you won’t be able to fix this problem. If you have mapped the FCoE VSAN onto a VLAN, then you only need to assign the vfc interface to the appropriate VSAN with these commands:

switch(config)# vsan database
switch(config-vsan-db)# vsan <number> interface vfc <number>
switch(config-vsan-db)# exit

At this point, the vfc interface will report up, and you should be able to see the host’s connection information with the show flogi database command.

From this point—assuming that your storage is attached to a traditional Fibre Channel fabric, which is likely to be the case in the near future—you only need to create zones with the WWNs of the FCoE-attached hosts in order to grant them access to the storage. Refer to my posts on creating zones and managing zones on a Cisco MDS for more information on this task.

In my own experience, once FCoE was properly configured on the Nexus 5000 switch, then creating zones and zonesets on the Cisco MDS Fibre Channel switch and creating and masking LUNs on the Fibre Channel-attached storage is very straightforward. This, as has been stated on several previous occasions, is one of the strengths of FCoE: it’s compatibility with existing Fibre Channel installations is outstanding.

Feel free to submit any questions or clarifications in the comments below.

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

Setting Up FCoE on a Nexus 5000

• New User’s Guide to Configuring Cisco MDS Zones via CLI
• New User’s Guide to Managing Cisco MDS Zones via CLI
• Why No Multi-Hop FCoE?
• FCoE and VLAN Trunking on Nexus 5000
• Enabling Jumbo Frames on a Nexus 5000

Let me see if I can more clearly articulate what I’m trying to say here. If you are a Twitter user and into virtualization or storage, then you probably are following either Chad Sakac of EMC (@sakacc on Twitter), Vaughn Stewart of NetApp (@vaughn_stewart on Twitter), or both. That being the case, you are probably very familiar with the extensive “discussions” that take place between the two of them. Both of them are very passionate about storage and virtualization, but they have differing viewpoints. Now, before I’m accused by NetApp of being an EMC bigot (which would be ridiculous given the coverage I’ve given NetApp) or accused by EMC of being a NetApp bigot (that, at least, might be understandable as I’m just now starting to learn EMC storage), let me say that I’m not endorsing either product. NetApp’s products and EMC’s products are different; each of them has strengths and weaknesses in different areas.

Now, ask yourself, “Why do these products have different strengths and weaknesses?” Do you know the answer? These products have different strengths and weaknesses because of the technology decisions each company chose to make in the products’ development. NetApp chose one path, EMC chose another. For NetApp, that has created certain efficiences, certain strengths—and corresponding weaknesses. Likewise, EMC’s technology decisions have resulted in their products having certain strengths and weaknesses. Neither of these products is perfect. For NetApp to claim that “their way is the right way” is ridiculous; their way is only one of many different ways to accomplish something. The same is true for EMC. And, by extension, the same is true for every other technology vendor on the planet.

You want more examples? Consider the architectural differences between VMware ESX/ESXi and Microsoft Hyper-V. The technology choices made by each company created inherent strengths and weaknesses in each product. VMware claims their choices are the best choices; Microsoft believes their architecture is the best. Clearly, neither product is perfect. Both products have their flaws.

The real key takeaway here is that no technology vendor has the right to throw rocks at another technology vendor. All technology vendors live in glass houses. For VMware to claim that Microsoft’s architecture is all wrong is, well, wrong. For EMC to say that NetApp’s technology choices are stupid would be wrong. For Xsigo to claim that FCoE is the wrong path for I/O virtualization is wrong (although, personally, I don’t consider FCoE an I/O virtualization technology, but that’s a different discussion for a different day). Why? Because every company has to make technology choices, and those technology choices will—by the very nature of technology—automatically create inherent differences, strengths, and weaknesses in the resulting product. And when you accept that truth (and it is a truth, I promise you), then you see why vendors should not engage in negative marketing. When a vendor engages in negative marketing about the competition, that vendor is simply inviting others to pick apart the flaws in their own products.

Of course, I’m not naive enough to believe that vendors will stop negative competitive marketing overnight. Still, I stand firm in the belief that those vendors that focus on the strengths of their products instead of the flaws of others’ products will move ahead. I’m certainly more likely to do business with them.

I’d be interested to hear what others have to say. Voice your position in the comments.

Disclosure: As you probably know, I work for a reseller who represents many different vendors and manufacturers. My words here are not endorsed by my employer, nor do I represent my employer in this area.

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

Cutting Yourself on the Double-Edged Sword

• Very High Quality vs. Just Good Enough
• I/O Virtualization and the Double-Edged Sword
• Storage Short Take #1
• UCS Training This Week
• Storage Short Take #3