Using this same definition, I would classify VPNs as a virtualization technology (they generally abstract the traffic inside the VPN tunnel from the network topology outside the tunnel). Wherever there is a layer of abstraction being inserted, then it’s valid to refer to that as virtualization.

Without a consistent definition, one could call server virtualization “running multiple workloads on a CPU” and ask how it was different from a general purpose operating system. Clearly, there are differences, but it’s possible to skew those differences depending upon how you define things.

I appreciate your comments and your insight, but I still stand by my statement that FCoE is not an I/O virtualization technology. An I/O technology, sure, but not an I/O virtualization technology because there is no layer of abstraction.

Is MR-IOV I/O virtualization? Again, it depends on how you define I/O.

You say FCoE cannot be I/O virtualization, because it is just FCP running on a shared wire.

One could say MR-IOV is not virtualization, because it is not virtualizing I/O, it is just I/O adapter sharing.

How is the end result different in these two cases?

1. A blade server chassis, where each server had an onboard CEE NIC, and a CEE switch in the blade chassis, connecting to an external CEE network.

2. A blade server chassis, where each server has no onboard NIC, but has an onboard PCI root port, and a multi-root aware PCI switch in the blade chassis, along with a multi-root aware CEE NIC in the blade chassis, connecting to an external CEE network.

Both provide virtualized, shared I/O to the individual servers.

The question is, which is better?

Layering SR-IOV operations at the individual blade server level on top of MR-IOV at the blade chassis level may create a far too complex, multi-layered virtualized PCI environment to manage.

Second, looking at some of the MR-IOV docs on the PCI SIG site, I see topics like switches, congestion management, buffers, performance monitoring … this is a network! A managed network, no less. A network with no mention of security, by the way (at least not that I can find).

Third, while SR-IOV is a technology which replaces existing software based adapter sharing mechanisms (NPIV, VMFS, vSwitch, Virtual IPs, etc.), MR-IOV seems like it may be a solution in search of a problem, unless its purpose is to replace a switch with an adapter, or more specifically, replace a many adapter/one switch solution with a one switch/one adapter solution.

Considering the latter, is what is being replaced (I/O adapters, or more specifically, Ethernet NICs) expensive? Are they underutilized, and therefore well suited for sharing?

Right now, some customers are scared of unified I/O because they feel they have too much I/O per server to put both storage and IP traffic onto a single 10Gb pipe. They want two dedicated 10GigE ports for IP and two dedicated 8Gb FC ports for storage. How would they accept 16 blade servers sharing a few NICs and HBAs, much less converged FCoE adapters?

PCIe MR-IOV looks a lot like NGIO, Future I/O, and the early days of InfiniBand: A switched I/O bus extension technology allowing sharing of I/O adapters. InfiniBand failed at this use, primarily because it inserted more complexity into the environment than the solution gained in efficiency. I do not see how MR-IOV will solve this.

You may remember a desire to use IB as a blade backplane I/O sharing technology. IBM’s BladeCenter H actually was designed with this capability. It may have made sense in the days of unvirtualized blade servers with two 1GHz cores and 2 GB RAM. Today, highly virtualized blades with eight 3 GHz cores, and 48 GB RAM (and 16 cores and 96 GB RAM in 2010), need enough dedicated I/O that adapter sharing makes little sense, in light of adding a new, intermediate, single-purpose network.

In essence, we need to build lossless ethernet fabrics that can support storage data in the same mode that fibrechannel uses - non blocking and lossless. Thus Converged Enhanced Ethernet (CEE) is the IETF standard that will create the underlying technology to build lossless ethernet fabrics for data centres while also supporting IP data that has different requirements.

However, the process of creating a lossless data centre network also is key enabler for iSCSI, since a lossless, low delay and highly available ethernet fabric will remove the technical limitations that stop iSCSI from high end deployment (put simply, and ignoring certain other topics)

Lossless Ethernet supports virtualization by enabling a single pair of 10GbE interfaces to act as both data and storage interface via a suitable CNA. The real question is whether customer choose FCoE CNA or an iSCSI CNA in the future ? Those with large legacy investments in FC storage will choose FCoE, however newer system may choose iSCSI to lower cost and increase network simplicity.

The major stumbling block is that Storage regards their network as inviolate and superior to the data network. I have two responses. One, FC networks are very, very small and don’t seem to scale past a few hundred ports. Two, thats exactly what telephony people said about their TDM networks (correctly) but are now fully integrated into the data networks as IP Telephony. Convergence is inevitable, history shows it always happens.

Note: Data Centre Ethernet is Cisco’s trademarked version of CEE, it may be either a superset of CEE or a proprietary version of CEE - the final outcome is not known. It should not be used when referring to the IETF or you are breaching trademark terms.

My understanding of MR-IOV is that this enables blade servers to change shape. Thus a blade of CPU’s, a blade of memory etc and you can cut your own hardware into pieces and then hand that to the hypervisor. Thats a quite different solution.

An MR-IOV switch, on the other hand, has the possibility of being something quite different. It will not be limited to Ethernet dependent protocols. It will communicate will communicate with servers over the PCIe bus. A robust MR-IOV-based TOR switch will be capable of housing diverse PCIe devices, such as 10GE NIC, FCoE CNA, FC HBA, IB, and even a device supporting some protocol we have yet to hear of (unlikely but possible). From the TOR, these interface cards, like the FCoE switch, will connect to a fabric and from there to other devices. The different is that the fabric doesn’t have to be an Ethernet fabric.

I’d just add that FCoE is much closer to reality than MR-IOV. But, now that MR-IOV is a PCIe standard, we’ll have to see what happens. Personally, I like the idea of uplinking high-density blade / 1RU servers with redundant external PCIe connectors to TOR MR-IOV switches and from there to some fabric. Just seems much more flexible than FCoE. We’ll have to wait and see if the ecosystem develops to support this vision.

Alternative opinions appreciated. Beyond the rack, there no difference as long as you are talking Ethernet.

FCoE is a halfway house kind of solution, designed to lessen the blow of a wholesale rip and replace of existing (expensive) FC infrastructure with 10GbE by taking a little chunk of Fiber out between the endpoint and the FCoE switch (I’m just calling it that for the sake of clarity - of course convergent switches will be the norm in the not too distant future) but leaving the rest of the Fiber infrastructure in place.

IOV on the other hand is more applicable to the convergent host adapter that can appear to the host as both an FC HBA and a 10GbE NIC at the same time.

Which is perfectly likely!