I’ve had this link sitting in my “Articles To Read” list for quite some time, but—to be perfectly honest—I’ve been just too busy to really do anything about it. Now that a hectic few weeks has wrapped up and I have a small breather before the next hectic few weeks, I wanted to comment briefly on Doug Gourlay’s discussion of FCoE versus MR-IOV.
First, some background: For those that aren’t familiar, FCoE is Fibre Channel over Ethernet, a T11 standard for running Fibre Channel Protocol over Ethernet, specifically 10 Gigabit Ethernet. More information on FCoE is found here. MR-IOV is Multi-Root I/O Virtualization, a PCI SIG specification for using PCI Express (PCIe) to connect and share multiple devices. More information on MR-IOV can be found here. MR-IOV is a multi-server extension to Single-Root I/O Virtualization, or SR-IOV.
Like Doug, I’ll put in a disclaimer that I haven’t read the report to which he’s referring in his article, either. However, as an individual who has done some research on the topic of I/O virtualization, I will say that anyone who compares FCoE to MR-IOV is comparing apples to oranges. These two technologies, in my mind, are designed to address two different problems.
FCoE provides the ability to use a single physical transport—10 Gigabit Ethernet, in this case—for Fibre Channel Protocol (FCP) as well as TCP/IP, iSCSI, and other Ethernet-borne protocols. This allows for the creation of a unified fabric, a single physical transport that carries all the various kinds of traffic that Ethernet-based Local Area Networks (LANs) and Storage Area Networks (SANs) carry separately today. Via the IETF Converged Enhanced Ethernet (CEE) standard—adopted by Cisco as Data Center EthernetTM—FCoE will ultimately have the same low, predictable latency and error-free operation that FCP enjoys today. FCoE is not, however, designed or architected to do anything other than allow FCP to run over Ethernet. It’s not intended to be a server interconnect technology. (Unless I’m missing something?)
MR-IOV, on the other hand, is intended to play in the server interconnect field. Its purpose is not to allow FCP to run over Ethernet, or to allow FCoE, iSCSI, and other TCP/IP protocols share the same physical connections. MR-IOV’s purpose is to allow multiple servers to share PCIe-based devices, like a FC Host Bus Adapter (HBA), or an iSCSI HBA, a 10 Gigabit Ethernet network interface card (NIC), or a video capture card. MR-IOV is intended to provide I/O virtualization, regardless of what type of I/O that might be. As long as the I/O runs across a PCI Express bus, MR-IOV comes into play.
I’ve heard multiple people refer to FCoE as an I/O virtualization technology, but I just don’t agree. FCoE only applies to FCP over Ethernet. It doesn’t apply to iSCSI. It doesn’t apply to video traffic, or audio traffic, or HTTP traffic. It only applies to FCP over Ethernet. While I might allow that FCoE does allow for a form of virtualization, by virtualizing the physical transport beneath FCP, I would not call it I/O virtualization. Further, FCoE and MR-IOV are complementary. You could use MR-IOV to share a single Converged Network Adapter (CNA), which provides FCoE and 10 Gigabit Ethernet functionality, among multiple servers. In this situation, what’s providing the I/O virtualization: MR-IOV, which is allowing multiple servers to use a single I/O card, or the CNA, which is putting the traffic onto the converged fabric?
I’m probably missing something huge here, some vital piece of information that would make sense why FCoE and MR-IOV would be considered competitive standards/specifications. Without that information, though, it just doesn’t make any sense to me to compare these two different yet complementary technologies. Someone want to enlighten me?
UPDATE: I’ve corrected my use of “Data Center Ethernet” to Converged Enhanced Ethernet (CEE) when referring to the IETF standard. As correctly pointed out in the comments, Data Center EthernetTM is a Cisco trademarked term referring to their implementation of CEE.
Tags: Cisco, FibreChannel, iSCSI, SAN, Standards, Storage, Virtualization
-
No, I’m with you. I think you are on the right track with this one. That is of course unless we are both wrong.
Which is perfectly likely!
-
Nope, you’re spot on as usual.
FCoE is a halfway house kind of solution, designed to lessen the blow of a wholesale rip and replace of existing (expensive) FC infrastructure with 10GbE by taking a little chunk of Fiber out between the endpoint and the FCoE switch (I’m just calling it that for the sake of clarity - of course convergent switches will be the norm in the not too distant future) but leaving the rest of the Fiber infrastructure in place.
IOV on the other hand is more applicable to the convergent host adapter that can appear to the host as both an FC HBA and a 10GbE NIC at the same time.
-
FCoE and MR-IOV are competitive technologies within the rack. They both want to become the next-gen TOR switch. Obviously, they’ll accomplish this in different ways. An FCoE switch, or to be more accurate an FCoE capable Ethernet switch, will be more of a traditional TOR switch–limited to protocols capable of being transported a top of Ethernet.
An MR-IOV switch, on the other hand, has the possibility of being something quite different. It will not be limited to Ethernet dependent protocols. It will communicate will communicate with servers over the PCIe bus. A robust MR-IOV-based TOR switch will be capable of housing diverse PCIe devices, such as 10GE NIC, FCoE CNA, FC HBA, IB, and even a device supporting some protocol we have yet to hear of (unlikely but possible). From the TOR, these interface cards, like the FCoE switch, will connect to a fabric and from there to other devices. The different is that the fabric doesn’t have to be an Ethernet fabric.
I’d just add that FCoE is much closer to reality than MR-IOV. But, now that MR-IOV is a PCIe standard, we’ll have to see what happens. Personally, I like the idea of uplinking high-density blade / 1RU servers with redundant external PCIe connectors to TOR MR-IOV switches and from there to some fabric. Just seems much more flexible than FCoE. We’ll have to wait and see if the ecosystem develops to support this vision.
Alternative opinions appreciated. Beyond the rack, there no difference as long as you are talking Ethernet.
-
My current interpretation is that FCoE is a transition standard that allows a smooth migration from Fibrechannel to iSCSI.
In essence, we need to build lossless ethernet fabrics that can support storage data in the same mode that fibrechannel uses - non blocking and lossless. Thus Converged Enhanced Ethernet (CEE) is the IETF standard that will create the underlying technology to build lossless ethernet fabrics for data centres while also supporting IP data that has different requirements.
However, the process of creating a lossless data centre network also is key enabler for iSCSI, since a lossless, low delay and highly available ethernet fabric will remove the technical limitations that stop iSCSI from high end deployment (put simply, and ignoring certain other topics)
Lossless Ethernet supports virtualization by enabling a single pair of 10GbE interfaces to act as both data and storage interface via a suitable CNA. The real question is whether customer choose FCoE CNA or an iSCSI CNA in the future ? Those with large legacy investments in FC storage will choose FCoE, however newer system may choose iSCSI to lower cost and increase network simplicity.
The major stumbling block is that Storage regards their network as inviolate and superior to the data network. I have two responses. One, FC networks are very, very small and don’t seem to scale past a few hundred ports. Two, thats exactly what telephony people said about their TDM networks (correctly) but are now fully integrated into the data networks as IP Telephony. Convergence is inevitable, history shows it always happens.
Note: Data Centre Ethernet is Cisco’s trademarked version of CEE, it may be either a superset of CEE or a proprietary version of CEE - the final outcome is not known. It should not be used when referring to the IETF or you are breaching trademark terms.
My understanding of MR-IOV is that this enables blade servers to change shape. Thus a blade of CPU’s, a blade of memory etc and you can cut your own hardware into pieces and then hand that to the hypervisor. Thats a quite different solution.
-
Is FCoE (or iSCSI, or InfiniBand) I/O virtualization? It depends on how you define I/O.
Is MR-IOV I/O virtualization? Again, it depends on how you define I/O.
You say FCoE cannot be I/O virtualization, because it is just FCP running on a shared wire.
One could say MR-IOV is not virtualization, because it is not virtualizing I/O, it is just I/O adapter sharing.
How is the end result different in these two cases?
1. A blade server chassis, where each server had an onboard CEE NIC, and a CEE switch in the blade chassis, connecting to an external CEE network.
2. A blade server chassis, where each server has no onboard NIC, but has an onboard PCI root port, and a multi-root aware PCI switch in the blade chassis, along with a multi-root aware CEE NIC in the blade chassis, connecting to an external CEE network.
Both provide virtualized, shared I/O to the individual servers.
The question is, which is better?
Layering SR-IOV operations at the individual blade server level on top of MR-IOV at the blade chassis level may create a far too complex, multi-layered virtualized PCI environment to manage.
Second, looking at some of the MR-IOV docs on the PCI SIG site, I see topics like switches, congestion management, buffers, performance monitoring … this is a network! A managed network, no less. A network with no mention of security, by the way (at least not that I can find).
Third, while SR-IOV is a technology which replaces existing software based adapter sharing mechanisms (NPIV, VMFS, vSwitch, Virtual IPs, etc.), MR-IOV seems like it may be a solution in search of a problem, unless its purpose is to replace a switch with an adapter, or more specifically, replace a many adapter/one switch solution with a one switch/one adapter solution.
Considering the latter, is what is being replaced (I/O adapters, or more specifically, Ethernet NICs) expensive? Are they underutilized, and therefore well suited for sharing?
Right now, some customers are scared of unified I/O because they feel they have too much I/O per server to put both storage and IP traffic onto a single 10Gb pipe. They want two dedicated 10GigE ports for IP and two dedicated 8Gb FC ports for storage. How would they accept 16 blade servers sharing a few NICs and HBAs, much less converged FCoE adapters?
PCIe MR-IOV looks a lot like NGIO, Future I/O, and the early days of InfiniBand: A switched I/O bus extension technology allowing sharing of I/O adapters. InfiniBand failed at this use, primarily because it inserted more complexity into the environment than the solution gained in efficiency. I do not see how MR-IOV will solve this.
You may remember a desire to use IB as a blade backplane I/O sharing technology. IBM’s BladeCenter H actually was designed with this capability. It may have made sense in the days of unvirtualized blade servers with two 1GHz cores and 2 GB RAM. Today, highly virtualized blades with eight 3 GHz cores, and 48 GB RAM (and 16 cores and 96 GB RAM in 2010), need enough dedicated I/O that adapter sharing makes little sense, in light of adding a new, intermediate, single-purpose network.
-
Virtualization is generally defined as “a layer of abstraction between computing elements.” Does FCoE add a layer of abstraction? Not as far as I am aware; hence, I do not refer to FCoE as a virtualization technology. Does MR-IOV? Yes, I believe–from what limited knowledge I have–that it does insert a layer of abstraction. SR-IOV does as well, to my knowledge. Thus, I would define these as virtualization technologies.
Using this same definition, I would classify VPNs as a virtualization technology (they generally abstract the traffic inside the VPN tunnel from the network topology outside the tunnel). Wherever there is a layer of abstraction being inserted, then it’s valid to refer to that as virtualization.
Without a consistent definition, one could call server virtualization “running multiple workloads on a CPU” and ask how it was different from a general purpose operating system. Clearly, there are differences, but it’s possible to skew those differences depending upon how you define things.
I appreciate your comments and your insight, but I still stand by my statement that FCoE is not an I/O virtualization technology. An I/O technology, sure, but not an I/O virtualization technology because there is no layer of abstraction.
Site Sponsors
7 comments
Comments feed for this article
Trackback link: http://blog.scottlowe.org/2008/11/17/fcoe-versus-mr-iovhuh/trackback/