FCoE

You are currently browsing articles tagged FCoE.

FCoE through a Nexus to a MDS-Attached Storage Array

Wednesday, December 15, 2010 in Interoperability, Storage by slowe | 11 comments

In this post, I want to pull together all the steps necessary to take a Converged Network Adapter (CNA)-equipped server and connect it, using FCoE, to a Fibre Channel-attached storage array. There isn’t a whole lot of “net new” information in this post, but rather I’m summarizing previous posts, organizing the information, and showing how these steps relate to each other. I hope that this helps someone understand the “big picture” of how FCoE and Fibre Channel relate to each other and how they interoperate (which, quite frankly, is one of the key factors for the adoption of FCoE).

The steps involved come from an environment with the following components:

A Dell PowerEdge R610 server running VMware ESXi and containing an Emulex CNA
A Cisco Nexus 5010 switch running NX-OS 4.2(1)N1(1).
A Cisco MDS 9134 Fibre Channel switch running NX-OS 5.0(1a).
An older EMC CX3-based array with Fibre Channel ports in the storage processors.

We’ll start at the edge (the host) and work our way to the storage. All these steps assume that you’ve already taken care of the physical cabling.

Depending upon how old the software is on your hosts, you might need to install updated drivers for your CNA, as I described here. If you’re using newer versions of software, the drivers will most likely work just fine out of the box.
The closest piece to the edge is the FCoE configuration on the Nexus 5010 switch. Here’s how to setup FCoE on a Nexus 5000. Be sure that you map VLANs correctly to VSANs; for every VSAN that needs to be accessible from the FCoE-attached hosts, you’ll need a matching VLAN. Further, as pointed out here, the VLAN and VLAN trunking configuration is critical to making FCoE work properly anyway.
The next step is connecting the Nexus 5010 to the MDS 9134 Fibre Channel switch. Read this to see how to configure NPV on a Nexus 5000 if you are going to use NPV mode (and read this for more information on NPV and NPIV). Using NPV or not, you’ll also need to setup connections between the Nexus and the MDS; here’s how to setup SAN port channels between a Cisco Nexus and a Cisco MDS.
Once the Nexus and the MDS are connected, then you’ll need to perform the necessary zoning so that the hosts can see the storage. Before starting the zoning, you might find it helpful to set up device aliases. After your device aliases are defined, you can create the zones and zonesets. This post has information on how to create zones via CLI; this post has information on how to manage zones via CLI.

At this point—if everything is working correctly—then you are done and you should be ready to present storage to the end hosts.

I hope this helps put the steps involved (all of which I’ve already written about) in the right order and in the right relationship to each other. If there are any questions, clarifications, or suggestions, please feel free to speak up in the comments.

Tags: Cisco, EMC, FCoE, FibreChannel, Interoperability, Networking, SAN, Storage

Technology Short Take #7

Monday, November 29, 2010 in Networking, Storage, Virtualization by slowe | 4 comments

Welcome to Technology Short Take #7! This time around I have a collection of links from networking, servers, storage, and virtualization. Our hot topics in this issue include Fibre Channel over Ethernet (FCoE) and its need—or lack thereof—for congestion management, Ubuntu on Hyper-V, the benefits of VAAI, and more!

Networking

I have a lot of FCoE-related links this time around. I’m not sure if that means FCoE has been getting more coverage or if it’s just a case of confirmation bias.

Need to decrypt a Cisco type 7 password? This page provides instructions on how it can be done. (Please be sure to use your powers for good, not evil.)
This blog post catalogue is a link list to a treasure trove of networking information.
I suppose this is one way of dealing with requests to do long-distance vMotion. I’m not so sure I agree that it’s an effective way.
The use of NIV to create the equivalent of multi-hop FCoE is something I discussed a while ago, but Brad Hedlund recently revisited it again. I can see both sides of the argument—both “for” and “against” considering fabric extenders as multi-hop FCoE—and I can see the need to use standard terminology to describe things. Without standard terminology, “multi-hop FCoE” means different things to different vendors and it’s hard for customers to make valid comparisons.
Erik Smith, a relatively new blogger, has a great introduction to FIP, FIP Snooping Bridges, and FCFs. If you’re new to FCoE—or even if you aren’t and want more detail—this is a great read with loads of relevant information. I’m looking forward to more of Erik’s posts on this topic.
The blog battle over FCoE’s need for QCN rages on. Joe Onisick does a good job of explaining QCN and why it might/might not be necessary, so if you’re unfamiliar with the debate that’s a good place to start. Ivan Pepelnjak breaks down 802.1Qau (the QCN standard) even further, providing more details on its operation and behavior. He then weights into the debate with this quick explanation and this comparison to Frame Relay. In the end, the answer to the question of FCoE’s need for QCN really boils down to everyone’s favorite IT answer: “It depends.” In this case, it depends upon your network design. With more DCB-capable switches between the end nodes and the FCFs, QCN becomes more valuable. With fewer (or no) DCB-capable switches between the end nodes and the FCFs, QCN offers far less benefit.

Servers

I’m adding this section because I have some articles that apply to servers, but not necessarily to virtualization. Since it fits in nicely with the data center theme of Technology Short Takes, it seems like a reasonable addition.

Jeff Allen, a UCS-focused CSE at Cisco, recently posted this guide to SAN boot with Cisco UCS. It’s definitely worth a read, especially if you’re new to UCS or haven’t done boot from SAN with UCS before.
I haven’t had nearly the time to blog about Cisco UCS as I would have liked, but Brian Gracely included me in this list of people to follow for Cisco UCS information. Thanks, Brian! I’ll do my best to earn my inclusion on the list.
Chris Fendya of WWT posted instructions on how to slipstream the Cisco UCS drivers into the installation of Windows Server 2003.

Storage

It’s funny to me that the storage section of these posts is typically the shortest. There are plenty of storage-related blogs out there, but almost all of them are high-level and tend not to provide the sort of down-to-earth “in the trenches” information I like to include. If readers have any suggestions for blogs that provide this sort of information, I’d love to hear them.

InformationWeek recently published this article on how to break free from Tier 1 SAN vendors. (Disclosure: I work for just such a Tier 1 SAN vendor.) I can’t say that I agree with the author’s reasoning; by the same token, customers should be able to go out and buy white box servers. Yet, companies such as HP and Dell are still selling lots of servers. Why is that? Because the value of a top-tier server is greater than the sum of its parts, and the same can be said for Tier 1 storage arrays. Now, having said that, I do agree that storage virtualization—which was the real focus of the InformationWeek article—can bring a lot of value and flexibility to the data center. I just don’t think that storage virtualization and Tier 1 storage arrays are mutually exclusive.
Here is a good “how to” on enabling ALUA and Round Robin multipathing with ESX and a CLARiiON CX4 array.
Bob Plankers has a great article on the impact of VAAI on storage operations. In this post, he shows how the write rate for his VAAI-capable HDS AMS 2500 drops to nothing when cloning templates. This is a great demonstration of how VAAI helps offload storage operations from the hosts to the array. Keep in mind that VAAI might not make operations faster, but it will make them more efficient. (It’s a subtle distinction, but an important one nevertheless.)
In the event you are considering pursuing CCIE Storage—a task that I’ve been strongly considering undertaking—Brian Feeny posted a list of CCIE Storage preparation resources.

Virtualization

If you’re thinking of installing Ubuntu Server on Hyper-V, you should have a look at Ben Armstrong’s recent write-up.
A few other Hyper-V links also made it into this week’s collection of links. First, we have a discussion of Hyper-V differencing disks with VDI. Second, we have another post by Ben Armstrong, this one containing a script to evacuate a Hyper-V cluster node on shutdown. Third, here’s a post on using Hyper-V dynamic memory to increase VDI density. (As an aside, I would still like to hear from Microsoft or another Hyper-V resource regarding dynamic memory. It still seems to me that without allowing memory overcommitment, dynamic memory is of limited value. Anyone?) Finally, Patrick O’Rourke posted some Hyper-V news from Tech-Ed Berlin; specifically, he was discussing the Hyper-V Cloud program. It’s nice to see Microsoft finally embracing the idea of the private cloud, validating what other virtualization competitors have been discussing for a while now.
Frank Brix aka “vFrank” posted more information on MAC address generation and address conflicts, including a list of the different VMware OUIs for MAC addresses.
Eric Sloof has a video on how to enable vsish, the VMkernel Sys Info Shell.
Frank Denneman has a good article on a potential issue with active/standby NICs and vSwitch failback. In his article, he describes a scenario (albeit a limited scenario) in which a cluster-wide isolation response could occur. The workaround is to use active/active NICs or set vSwitch failback to No.
Frank also posted a follow-up article on the topic of NUMA, hyperthreading, and the numa.preferHT advanced setting.
Gavin Adams has a good “how to” on replacing the vCenter Server SSL certificate. It might be worth a read if you are in a situation that requires a valid SSL certificate on your vCenter Server system.
This article is a good overview of VMware vShield, including Zones, App, Edge, and Endpoint.
Several interesting VMware KB articles found their way to my URL Inbox, including this one on changing the IP address of the vCloud Director server; this KB article on an issue with VMXNET3 virtual adapters and cloning Windows 2008 R2 or Windows 7 templates; this one on an issue with the in-box Broadcom bnx2x driver with ESX/ESXi 4.1; and an article on an HA error with vCenter 4.1 and ESXi 4.0. It also looks like Hany Michael has started contributing diagrams to the VMware KB: see here and here.
Eric Sloof provides more information on using esxtop to measure SCSI reservations and reservation conflicts; see this post and this follow-up post.
While this is a bit old (published back in late September), Dave Hill has a post on a building block design for VMware vCloud Director.

That wraps up this installation of Technology Short Takes. As always, your comments, thoughts, suggestions, or clarifications are welcome, so please speak up in the comments!

Tags: Cisco, FCoE, HyperV, Microsoft, Networking, Security, Storage, Virtualization, vSphere

Technology Short Take #4

Tuesday, September 28, 2010 in Networking, Storage, Virtualization by slowe | No comments

Welcome to Technology Short Take #4, the latest collection of virtualization, storage, networking, and other data center-related links, news, thoughts, and views. As always, I hope you find something interesting here!

First up, we’ll revisit the idea of multi-hop FCoE. You might recall a couple of articles I wrote a while ago about Network Interface Virtualization (NIV) and the role of NIV in FCoE. One of my new favorite networking bloggers, Ivan Pepelnjak, picked up this same topic recently in this post on multi-hop FCoE. (Did I also link to his earlier multi-hop FCoE piece?) Anyway, it’s a good read and worth reviewing if you’re at all unclear about how these various technologies play together.
William Lam—whose Virtually Ghetto blog joined the Top 25 Virtualization Blogs in the last round of voting—has published a primer on VMware vsish, the VMkernel System Interface Shell. William has done an awesome job of providing a great level of detail and information about this previously undocumented utility. Great work, William, and congratulations on your top 25 blog position!
Frank Denneman has posted another great in-depth technical article on how the ESX/ESXi CPU scheduler handles NUMA nodes and wide VMs. This is seriously good stuff and should be on the must-read list for anyone wanting to better understand how ESX/ESXi works. Frank also had a post on resource pools and simultaneous vMotions that is worth reading. Yet another reason not to use resource pools as a folder structure!
Fellow co-worker Aaron Delp brought to light a key design consideration: how should a VMware architect handle 10GbE and vSphere 4.1′s increased vMotion throughput? His article (available here) triggered a number of other articles, such as this article on VMware QoS designs with Cisco UCS and Nexus by noted networking expert Brad Hedlund. I love it when a fellow blogger triggers a great conversation like this. Good stuff!
Noted VMware performance guru and (now) vSpecialist Scott Drummonds recently posted a great piece on optimizing vSphere for hyper-threading. In his article, Scott discusses the NUMA.preferHT configuration parameter and the potential implications of using that parameter. Also be sure to check out Scott’s article on databases, storage, and solid state disks. This is just another example of needing to take a holistic approach to performance and how new technologies (like EMC FAST in this article) are affecting designs.
Justin Guidroz had a good post with a script aimed at helping update the IPMI settings on ESX hosts with the settings from UCS Manager. Why is this important? IPMI is the mechanism whereby VMware vSphere can power servers down as part of VMware Distributed Power Management (DPM), and out-of-sync settings can prevent DPM from working properly.
If you’ve ever wanted to better understand mirror positions in EMC’s Symmetrix arrays, fellow vSpecialist Travers Nicholas has a good write-up here. Mirror positions are a key part of how the Symmetrix operates, and Travers does a good job of explaining the basics.
On the Hyper-V side of the blogging world, I’ve seen a few interesting posts come from Ben Armstrong (aka Virtual PC Guy), who appears to be on something of a scripting kick. First there’s this article on setting up non-administrative control of Hyper-V, followed up by creating a Hyper-V Administrators local group through PowerShell. Oh, and Ben continued his series on working with Hyper-V dynamic memory with this post on changing the minimum memory.
If you are using VM failure monitoring with VMware HA, this recent VMware KB article shows you how to determine if a VM reboot was caused by VMware HA.

I guess that’s about it for this time around. Before I go, though, here are a few miscellaneous links you might find interesting:

10 Networking Books to read before you die
EMC Replication Manager With vSphere 4 and LVM.EnableResignature
RecoverPoint in Vblock for SAP
Error Upgrading VMware Tools vSphere 4.1
Automating ESXi 4.1 Kickstart Tips & Tricks

As always, feel free to share other links or items in the comments below. Thanks for reading this far!

Tags: Cisco, EMC, FCoE, HyperV, Microsoft, Networking, Storage, Symmetrix, Virtualization, VMware, vSphere

Technology Short Take #3

Friday, September 10, 2010 in Microsoft, Networking, Storage, Virtualization by slowe | 4 comments

Welcome to Technology Short Take #3, a collection of links about key data center technologies like virtualization, networking, and storage. I’m still striving to broaden the scope of these posts to include even more storage and networking posts, so I’d love to hear feedback from readers on how well I’m doing and what other sources I should consider for inclusion here.

But enough of that for now; on with the content!

Priority Flow Control (PFC) is an as-yet-unratified IEEE standard (IEEE 802.1Qbb) that is often linked closely to Fibre Channel over Ethernet (FCoE). If you’re interested in getting a bit more information on PFC and you’re not (yet) a networking expert, this introduction to 802.1Qbb is pretty handy.
Didier Pironet recently documented some of vSphere 4.1′s advanced iSCSI settings. Good information, although what would be really handy was any recommendations around whether changing any of these settings should be considered and in what environments a change might be recommended. A future post, Didier?
Ben Armstrong (aka “The Virtual PC Guy”) has posted two articles so far dealing with how to script Hyper-V’s dynamic memory. Part 1 shows how to read the dynamic memory configuration; part 2 shows how to display the current usage information. Ben also recently published an article on parent memory reserve, which is how Hyper-V reserves money for the parent partition running Windows Server 2008.
This SearchTelecom.com article on Locator/ID Separation Protocol (LISP) gave me just the introduction I needed to LISP. After you read that article, you can continue your LISP education by visiting this brief blog post and checking out some of the other linked resources.
I’ve written before about multi-hop FCoE (here and here, for example), but this post on multi-hop FCoE 101 is a great read and highlights some of the differences in vendor implementations. Based on the article, it’s these differences in vendor implementations that often lead to disagreements between the vendors with regard to what’s required or not required. (It seems like I’m really digging some of Ivan’s stuff recently. I’m going to have to add him to my list of networking RSS feeds!)
Tried SLES 11 SP1 for VMware yet? Jase McCarty recently took it for a quick spin. Here are his thoughts.
Jason Boche recently posted a fix for an error with vCenter Service Status in vCenter Server 4.1.
For those readers who haven’t had the opportunity to work with Cisco’s Unified Compute System (UCS), there are lots of great bloggers out there writing about it—too many to name, in fact. Kevin Goodman captured a few of them in this list of Cisco UCS people and blogs. While you’re coming up to speed, though, this page from Cisco on upgrading the BIOS on a Cisco UCS server blade gives you an idea of how the system uses service profiles as the vehicle for almost everything. This similar post on Cisco’s web site breaks down the process of creating a service profile in UCS, a topic that I’ve tackled myself (with a four-part series that starts here).
This page listing the Cisco UCS B-series network adapters shows some “Gen 2″-type cards, such as the M72KR-E Emulex CNA. Did I miss an announcement? Unlike the “Gen 1″ cards, the “Gen 2″ cards aren’t hyperlinked for more information.
William Lam (@lamw on Twitter and elsewhere), whom I had the great pleasure of meeting personally last week while at VMworld 2010, has published what is likely to be the definitive primer on vsish, a largely undocumented utility. Check out the vsish write-up on William’s site. I also recently found an older article that William wrote on how to remove stale targets from vMA. vMA-related articles are almost like gold these days since all the geeks are needing a new command-line fix for ESXi.
Working on a VDI environment and want to disable some of the welcome stuff that Windows throws your way? Check this out.

There were a few other links that I collected as well but didn’t really have anything to say about them; still, in the event they might prove useful, here they are:

Krystaltek: What ESX Admins Group? – A Tale of RTSM and AD
vCenter SRM Automatic Failback Options Using EMC Storage
Support Insider: VMware Snapshots
Running the VMware vSphere Hypervisor stateless
Best practice in LUN design (VMware Communities)
VMware KB: Using a VNC Client to Connect to Virtual Machines
VMware KB: Do I choose the PVSCSI or LSI Logic virtual adapter on ESX 4.0 for non-IO intensive workloads?
VMware ESXi 4.1 does not use whole disk capacity for VMFS3

That should do it this time around. Thanks for reading, and feel free to suggest additional articles or links that you think other readers would find useful.

Tags: FCoE, HyperV, Microsoft, Networking, Virtualization

How to Configure NPV on a Nexus 5000

Monday, June 28, 2010 in Networking, Storage by slowe | 14 comments

Some time ago I posted a “how to” article on how to configure FCoE on a Nexus 5000 switch. At that time, I did not put the Nexus 5000 into NPV mode but rather connected it to a Cisco MDS Fibre Channel switch without using NPV. In this entry, I’d like to follow up on that article and show you how to configure NPV on a Nexus 5000.

If you aren’t familiar with NPV (N_Port Virtualization) and how it’s different than NPIV (N_Port ID Virtualization), check out this article titled “Understanding NPIV and NPV”; It should help clear things up. Because NPV makes the Nexus 5000 look like an NPIV-enabled host, one potential use for NPV, as in this case, is when connecting the Nexus 5000 to a non-Cisco Fibre Channel switch. Using NPV helps ease interoperability concerns. In this instance, I’m connecting the Nexus 5000 to a Brocade Fibre Channel switch (actually an EMC Connectrix).

Note that I tested these instructions on a Nexus 5010 using both NX-OS 4.1(3)N2(1) as well as NX-OS 4.2(1)N1(1).

The first step is to enable NPV on the Nexus 5000. As far as I can tell, in order to enable NPV you must first enable FCoE using the feature command:

switch(config)# feature fcoe

This loads various Fibre Channel modules and makes possible other features, including NPV and NPIV. Enabling NPV erases the switch configuration and reboots the switch, so be sure you are connected via a console connection before enabling NPV with the feature command:

switch(config)# feature npv

Immediately after enabling NPV, the Nexus 5000 will reboot (you’re warned and given the option to proceed or cancel). The warning indicates that the switch’s configuration will be removed, but the minimally-configured switches I used in my testing retained their configuration. Granted, I hadn’t performed any Fibre Channel or FCoE configurations yet, so perhaps that’s the configuration to which the warning was referring.

Once NPV is enabled on the switch, you can then configure Fibre Channel uplink ports as NP ports (proxy N_ports); these are also referred to as external interfaces. To configure a Fibre Channel port as an NP port, use these commands:

switch# config t switch(config)# interface fc slot/port switch(config-if)# switchport mode np switch(config-if)# no shut switch(config-if)# exit switch(config)# exit

You should then be ready to physically connect to the upstream Fibre Channel switch, which—if you recall correctly from my earlier NPV/NPIV post—needs to be NPIV-enabled. In this particular case, I was uplinking the Nexus 5010 to an EMC-rebranded Brocade switch (a Connectrix DS-300B running Fabric OS 6.1.0). To show whether the port on the Connectrix was enabled for NPIV, I used the portcfgshow command:

rtp-fc-sw-01:admin> portcfgshow port number

Look for the line that says “NPIV Capability”; the value should be reported as “ON”. If the value is not “ON”, you’ll need to use the portcfgnpivport command to enable NPIV on the specified port, like this:

rtp-fc-sw-01:admin> portcfgnpivport port number 1

The “1″ at the end of that command enables NPIV; a “0″ would disable NPIV for that port.

Once NPIV is enabled on the upstream Fibre Channel switch, when you physically connect the configured external interface then the Fibre Channel link should come up. I used the show int fc slot/number command on the Nexus to verify that the port was up; on the Connectrix, I used the portshow port command. In addition, I was also able to see the Nexus switch logged into the Fibre Channel fabric on the Connectrix using the nsshow command.

Once you have Fibre Channel connectivity via the external interfaces, then configuring FCoE to hosts connected to the Nexus follows the same set of instructions laid out in my earlier FCoE how-to article.

From that point on, it’s only a matter of configuring zones (see here for help with zones on a Cisco MDS) and presenting storage. Those are different posts for a different day…

Tags: Cisco, FCoE, FibreChannel, Nexus, NPIV, NPV, Storage

A Collection of FCoE Posts

Tuesday, May 4, 2010 in Storage by slowe | No comments

I’ve had these FCoE-related articles sitting around in my Yojimbo database for a while, but I’m only now getting around to doing something with them. There’s some great information in these posts, but be sure to check the comments to the posts as well—there’s some equally good information to be found there as well.

FCoE Multi-hop: Why wait?
Re-examining FCoE and iSCSI Pros and Cons
FCoE vs. iSCSI: The Cagefight!
Gartner on FCoE. Whoa There, Sparky
8Gb Fibre Channel or 10Gb Ethernet w/ FCoE?

Tags: FCoE, FibreChannel, iSCSI, Storage

Thinking Out Loud: What is multi-hop FCoE?

Wednesday, April 21, 2010 in Networking by slowe | 10 comments

It’s been a while since I last posted a “Thinking Out Loud” post, but a Twitter discussion this morning prompted me to post this one. Last night, I posted a question to Brad Hedlund (@bradhedlund on Twitter and a great data center networking resource) regarding the Nexus 2232PP fabric extender. My tweet to Brad was this: does the Nexus 2232PP make “multi-hop FCoE” possible via NIV?

(If you haven’t already read my post on the relationship between FCoE and NIV, go read it now as it provides a good background and context for this discussion.)

Brad responded, confirming my assessment and stating that FIP snooping wasn’t necessary. I wasn’t sure about the FIP snooping part but after seeing the response I now understand. In any event, a number of others starting questioning my use of “multi-hop FCoE” in conjunction with the 2232PP fabric extender, stating that it wasn’t a hop because ti does not actually switch any traffic.

Strictly speaking, all of these individuals are absolutely correct; for more information, go read this post on NIV. In this case, the Nexus 5000 is the interface virtualization (IV)-capable bridge and the Nexus 2232PP is the interface virtualizer (IV). The IV doesn’t switch any traffic; all switching occurs on the IV-capable bridge. Therefore, from a switching perspective, a Nexus 5000 and any or all associated fabric extenders are a single hop.

My response is this: what is multi-hop, anyway? Is it the presence of multiple switches in a data path between an initiator and a target? Or is it the presence of multiple physical devices, chained together, between an initiator and a target? In the first definition, using a fabric extender isn’t multi-hop; in the second definition, it is. More to the point, should a customer really care which definition is correct? Why is multi-hop FCoE important, anyway? I would say the answer to that question is scalability. Customers want to be able to scale the FCoE fabrics larger than they can today. Multi-hop FCoE—however you want to define it—makes that possible. So does it really matter how we get there? Besides which point, using the fabric extender approach means only a single point of management instead of multiple points of management. Isn’t that better?

What do you think? Do your own “thinking out loud” in the comments below.

Tags: Cisco, FCoE, Networking, Nexus, ToL

Am I Understanding RoCE Correctly?

Tuesday, April 20, 2010 in Networking by slowe | 8 comments

A couple of days ago I posted a tweet inquiring about RDMA (Remote Direct Memory Access) over Converged Ethernet, affectionately known as RoCE and even more affectionately pronounced “Rocky”. At the time I was unclear exactly what RoCE was and what it was trying to accomplish.

Since then, I’ve done a bit more research and I think that I have a better idea of RoCE now. In particular, this EE Times article provided some information that I found useful in putting the pieces together.

If I understand things correctly, RoCE does for InfiniBand what FCoE did for Fibre Channel—it replaces the physical transport mechanism with 10 Gigabit Ethernet. More specifically, Converged Ethernet, which is the particular flavor of 10Gb Ethernet that supports the IEEE Data Center Bridging (DCB) standards like Priority-Based Flow Control (802.1Qbb), Enhanced Transmission Selection (802.1Qaz), and Congestion Notification (802.1Qau). These DCB standards are intended to make Ethernet less “lossy” and “chatty” and more reliable, predictable, and lossless like Fibre Channel (or InfiniBand).

Fibre Channel over Ethernet (FCoE) took the physical transport layers of traditional Fibre Channel and replaced them with Ethernet, and the IEEE created the DCB efforts to make sure that the underlying Ethernet transport was reliable and lossless so that it could support FCoE. Because it still “looked” like Fibre Channel at the upper layers, there is a great deal of interoperability between Fibre Channel and FCoE.

In a similar fashion, RDMA over Converged Ethernet (RoCE) does the same sort of thing, but for the RDMA interfaces that are common to InfiniBand. It takes RDMA and puts it on Ethernet, again relying upon the IEEE DCB standards to make Ethernet reliable and lossless with predictable latencies. No more proprietary fabrics; with RoCE-capable adapters, you’ll be able to reap the ultra-low latency benefits of InfiniBand over standards-based 10Gb Converged Ethernet.

At least, that’s how I understand it. Anyone else have a better explanation?

Tags: FCoE, Networking

How NIV and FCoE Play Together

Sunday, March 21, 2010 in Networking, Virtualization by slowe | 15 comments

Last year, I wrote a piece about multi-hop Fibre Channel over Ethernet (FCoE) and some of the various reasons why—at the time—multi-hop FCoE was not a practical reality. Some things have changed since I last discussed multi-hop FCoE, and today I’d like to take a quick look at the interaction between network interface virtualization (NIV) and FCoE and see how this plays into multi-hop FCoE.

If you haven’t already read my recent article on understanding NIV, go read it now. Likewise, if you haven’t read the multi-hop FCoE article, you should go read it now too. Believe me, the rest of this article will make a lot more sense if you do.

Done now? Good. Let’s get started.

From my article on multi-hop FCoE, I identified two key roadblocks to multi-hop FCoE support:

First, there was a lack of widespread support for FCoE Initialization Protocol (FIP). FIP allows an FCoE initiator to be separated from a Fibre Channel forwarder (FCF) by one or more IEEE DCB-capable switches. This has since been addressed by updates to NX-OS (the code that runs on Cisco’s Nexus 5000 switches) and by Generation 2 converged network adapters (CNAs); both of these components now have FIP support included.
Second, there was no defined standard for creating the FCoE equivalent of trunking E_ports (which I believe are referred to as VE_ports). VE_ports are necessary to link multiple FCFs together, much in the same way you would use an inter-switch link (ISL) in traditional Fibre Channel storage area networks (SANs). To my knowledge, this issue has not yet been addressed.

At first glance, NIV doesn’t seem to help with either of these roadblocks. When you take a deeper look, though, you’ll see that NIV can actually serve as a workaround for both problems. Here’s how.

In NIV, recall that you have both an interface virtualization (IV)-capable bridge (or switch) as well as one or more interface virtualizers (IVs). (Remember that IVs are also referred to as fabric extenders.) Network interface cards (NICs) connect to ports on the IVs, and the IVs uplink to the IV-capable bridge. Even though the IV-capable bridge and the IVs are physically separate devices, they appear as a single device. Even though there is a connection—typically an Ethernet connection—between the IVs and the IV-capable bridge, it appears and functions as a single device.

With this in mind, then, I’ll ask this question: what is a multi-hop topology? If a multi-hop topology is multiple physical devices connected over an Ethernet uplink, then multi-hop FCoE is possible today with an FCoE-enabled IV-capable bridge and one or more FCoE-enabled IVs. In fact, this is the topology that Cisco uses in its Unified Computing System (UCS): a pair of FCoE-enabled IV-capable bridges (the UCS 6100XP fabric interconnects) connected to one or more FCoE-enabled IVs (the I/O Modules in the back of the chassis).

Applying this line of thinking to our roadblocks above, we see that the use of NIV allows for greater port densities; greater port density is one of the primary reasons why users would want FCoE initiators separated from an FCF by an IEEE DCB-capable switch. In addition to leveraging FIP (and eventually leveraging the IEEE DCB standards once they are finalized), you can build the same sort of topology using an IV-capable bridge and one or more IVs.

Similarly, using NIV as a way of connecting multiple devices together eliminates the need to chain multiple FCFs together; this, in turn, eliminates the need for the FCoE equivalent of ISLs and the need to create VE_ports between the FCFs. So NIV helps to address the second roadblock as well.

Of course, NIV isn’t the only way the industry is going to address the need for multi-hop FCoE. Further—to my knowledge, at least—NIV is a Cisco-only approach. As FCoE continues to mature and the IEEE DCB standards are finalized and ratified, then organizations can leverage a standards-based approach to building more complex FCoE topologies than are currently possible today.

Courteous comments, corrections, and thoughts (with full vendor disclosure, please) are welcome below.

Tags: Cisco, FCoE, Networking, Virtualization

Storage Short Take #5

Tuesday, November 17, 2009 in Storage by slowe | 6 comments

I’ve decided to resurrect my Storage Short Take series, after almost a year since the last one was published. I find myself spending more and more time in the storage realm—which is completely fine with me—and so more and more information coming to me in various forms is related to storage. While I’m far from the likes of storage rockstars such as Robin Harris, Stephen Foskett, Storagebod, and others, hopefully you’ll find something interesting and useful here. Enjoy!

This blog post by Frank Denneman on the HP LeftHand product is outstanding. I learned more from this post than a lot of posts recently. Great work Frank!
Need a bit more information on FCoE? Nigel Poulton has a great post here (it’s a tad bit older, but I’ve just stumbled across it) with good details for those who might not be familiar with FCoE. It’s worth a read if you haven’t already taken the time to come up to speed on FCoE and its “related” technologies.
What led me to Nigel’s FCoE post was this post by Storagezilla in which he rants about “vendor flapheads” who “are intentionally obscuring it’s [FCoE's] limitations”. You’ve got that right! Wanting to present a reasonably impartial and complete view of FCoE was partially the impetus behind my end-to-end FCoE post and the subsequent clarification. Thankfully, I think that the misinformation around FCoE is starting to die down.
This post has a bit of useful information on HP EVA path policies and vSphere multipathing. I would have liked a bit more detail than what was provided, but the content is good nevertheless.
Devang Panchigar’s recoup of HP TechDay day 1, which focused on HP StorageWorks technologies, has some good information, especially if you aren’t already familiar with some of HP’s various storage platforms.
Chad Sakac of EMC has some very useful information on Asymmetric Logical Unit Access (ALUA), VMware vSphere, and EMC CLARiiON arrays. If you using EMC storage with your VMware vSphere 4 environment, and you have a CX4, and you’re running FLARE 28.5 or later, it might be worthwhile to switch your path policy from NMP to Round Robin (RR).
Speaking of RR with vSphere, somewhere I remember seeing information on changing the default number of I/Os down a path, and tweaking that for best performance. Was that in Chad’s VMworld session? Anyone remember?
If you’re looking for a high-level overview of SAN and NAS virtualization, this InfoWorld article can help you get started. You’ll soon want to delve deeper than this article can provide, but it’s a reasonable starting point, at least.