Continuing the FCoE Discussion

A few weeks ago I examined FCoE in the context of it’s description as an “I/O virtualization” technology in my discussion of FCoE versus MR-IOV. (Despite protestations otherwise, I’ll continue to maintain that FCoE is not an I/O virtualization technology.)

Since that time, I read a few more posts about FCoE in various spots on the Internet:

Is FCoE a viable option for SMB/Commercial?
Is the FCoE Starting Pistol Aimed at iSCSI?
Reality Check: The FCoE Forecast

Tonight, after reading a blog post by Dave Graham regarding FCoE vs. InfiniBand, I started thinking about FCoE again, and I came up with a question I want to ask. I’m not a storage expert, and I don’t have decades of experience in the storage arena like many others that write about storage. The question I’m about to ask, then, may just be the uneducated ranting of a fool. If so, you’re welcome to enlighten me in the comments.

Here’s the question: how is FCoE any better than iSCSI?

Now, before your head explodes with unbelief at the horror that anyone could ask that question, let me frame that question with more questions. Note that these are mostly rhetorical questions, but if the underlying concepts behind these questions are incorrect you are, again, welcome to enlighten me in the comments. Here are the framing questions that support my primary question above:

  1. FCoE is always mentioned hand-in-hand with 10 Gigabit Ethernet. Can’t iSCSI take advantage of 10 Gigabit Ethernet too?
  2. FCoE is almost always mentioned in the same breath as “low latency” and “lossless operation”. Truth be told, it’s not FCoE that’s providing that functionality, it’s CEE (Converged Enhanced Ethernet). Does that mean that FCoE without CEE would suffer from the same “problems” as iSCSI?
  3. If iSCSI was running on a CEE network, wouldn’t it exhibit predictable latencies and lossless operation like FCoE?

These questions—and the thoughts behind them—are not necessarily mine alone. In October Stephen Foskett wrote:

And iSCSI isn’t done evolving. Folks like Mellor, Chuck Hollis, and Storagebod are lauding FCoE at 10 gigabit speeds, but seem to forget that iSCSI can run at that speed, too. It can also run on the same CNAs and enterprise switches.

If those Converged Network Adapters (CNAs) and enterprise switches are creating the lossless CEE fabric, then iSCSI benefits as much as FCoE. Dante Malagrino agrees on the Data Center Networks blog:

I certainly agree that Data Center Ethernet (if properly implemented) is the real key differentiator and enabler of Unified Fabric, whether we like to build it with iSCSI or FCoE.

Seems to me that all the things that FCoE has going for it—10 Gigabit speeds, lossless operation, low latency operation—are equally applicable to iSCSI as they are functions of CEE and not FCoE itself. So, with that in mind, I bring myself again to the main question: how is FCoE any better than iSCSI?

You might read this and say, “Oh, he’s an FCoE hater and an iSCSI lover.” No, not really; it just doesn’t make any sense to me how FCoE is touted as so great and iSCSI is treated like the red-headed stepchild. I have nothing against FCoE—just don’t say that it’s an enabler of the Unified Fabric. (It’s not. CEE is what enables the Unified Fabric.) Don’t say that it’s an I/O virtualization technology. (It’s not. It’s just a new transport option for Fibre Channel Protocol.) Don’t say that it will solve world hunger or bring about world peace. (It won’t, although I wish it would!)

Of course, despite all these facts, it’s looking more and more like FCoE is VHS and iSCSI is Betamax. Sometimes the “best” technology doesn’t always win…

Tags: , , ,

  1. Justin’s avatar

    Scott,
    Please forgive me if I’m explaining stuff you’ve heard before and am insulting your intelligence, but…

    From my experience the big deal with FCoE vs. iSCSI is that one is working at Layer 2 and one is working at Layer 3 (respectively). iSCSI is SCSI commands and data (1), encapsulated in TCP packets (2), sent using IP (3), over ethernet frames (4). FCoE is SCSI commands and data (1), encapsulated in FC frames (2), sent over ethernet frames (3). It’s just inherently less overhead.

    FCoE is essentially SCSI over Ethernet, whereas iSCSI is SCSI over IP… Similarly, there does exist FCoIP, which is comparable to iSCSI although it’s used for a completely different purpose (tunneling FC from site to site).

    So there’s less work to do for FCoE, and this means less work to do either in the software driver for it, or in the adapter. Also of note is that today ,a hardware iSCSI adapter will run you about the same as a FC HBA, so there’s not that much cost savings. The other important point is that FCoE will work with existing FC storage arrays with no modification, which means you can start your unified fabric while still maintaining your old FC infrastructure.

    I do think that 10Gbit iSCSI will be better than 1GB iSCSI, obviously, but FCoE will be better. There will be support for existing enterprise storage arrays in much greater abundance with FCoE than iSCSI, and your performance will be better straight off the bat (a 4GB FC array via FCoE will be way faster than a 1GB iSCSI array on 10GBe).

    That being said, iSCSI will always be routable whereas FCoE won’t be… but do you really want to be routing your storage traffic?

    I think the market will eventually decide which technology will win, and just from the way I see it I’m betting on FCoE due to its compatibility.

    Your thoughts?

  2. Rodos’s avatar

    Great topic Scott. There is a thread on VMTN about this at the moment and it would be great if people who are thinking about this would be able to contribute. Its at http://communities.vmware.com/message/1119046

    Rodos

  3. Rodos’s avatar

    FCoE is better than iSCSI when you need to integrate into existing FC fabrics that may already exist.

    Its a fruit comparison but not an apples to apples. One use case is if you have your existing FC storage fabric but want to bring in new equipment, you can use FCoE for that at the access layer and then transport it over FC in the core to get to the existing SANs. Not many SANs natively support FCoE, but yes they do support iSCSI. However many don’t allow FC and iSCSI either at the same time or to the same LUNs. But my point is more about transition and integration to existing environments.

    Just a thought as to one difference.

  4. Gary’s avatar

    I see where the confusion comes here. The main difference between iSCSI and FCoE is that iSCSI uses the TCP/IP stack and FCoE runs directly on ethernet. Ethernet has no flow control, therefore the new converged switches need to add flow control to the traffic (bit of a wikipedia lookup answer, but the interpretation is import).

    Like you I am not a storage guru, but as far as the other main activities that are associated with FC, zoning and mapping, the new converged switches will need to perform this process. iSCSI does not have these facilities, although they can be mimicked using access lists. FCoE is not routable, unlike iSCSI, so all devices from initiator to target need to be converged (FCoE aware) devices to create a fabric.

    The nice thing about iSCSI is that it can be implemented on an IP network. It’s performance is going to be tied into the setup and any hardware used to provide the transport (doing iSCSI on hardware, rather than in software provides benefits as well as tech like jumo frames). iSCSI will also inherit any issues with the IP network it is running on top of.

    FCoE is going to add an additional cost to the infrastructure at any site in which it is deployed. Converged switches will not be cheap, and compatible HBAs will still be required. Whether the HBA will also be converged is another question (can I run FCoE and TCP/IP over the same port?).

    The bit of knowledge I am missing is how converged are these devices? Once the initiator connects to the converged switch, is the transport between these switches also converged (holding both TCP/IP networking and FCoE traffic), even further then if this is the case then how do the storage admins feel about their storage path being poluted with other traffic?

    I would almost consider that FCoE only really provides a few benefits
    1. Brings FC to 10gig
    2. Reduces the number of deployed devices (network + fabric vs. converged)
    3. Changes the medium to a lower cost alternative and enables storage pathing over infrastructure that might be more readily available (gets rid of fibre runs and replaces with copper)

    I’m probably wrong about a lot of this stuff, so some clarification would help both myself and other readers of Scott’s (excellent) blog.

    Gary

  5. Gary’s avatar

    The difference between iSCSI and FCoE is mostly down to the transport. FCoE uses ethernet as the transport and iSCSI uses TCP/IP. Ethernet needs to be extended to support flow control (already part of the the TCP/IP standard).

    FCoE requires a converged device to perform the following
    1. Map MAC address to WWW
    2. Perform zoning/mapping/masking functions
    3. Create fabric between initiator and target

    The nice thing about iSCSI is that the network doesn’t need to change, standard switching and routing can provide a route from initiator to target (routing being the key word, as ethernet is not routable). iSCSI does not provide the zoning/mapping/masking functions, but some of this can be achieved via access lists and clever VLAN tagging.

    FCoE also supports VLAN tagging, so logical separation of traffic is still possible (rather than the guaranteed physical separation fabric provies).

    Adapters will also be converged, so both TCP/IP and FCoE can use the same medium. This is were I think the standard helps some implementations, but hinders others. here’s my thinking.

    If you converge the two transports then you need to have some kind of QOS in place to insure the storage path is not interrupted. Storage admins like to know where the bottlenecks can exist in the transport (fabric) to identify throughput issues. The converged devices will need a management system for both QOS and throughput analysis to satisfy the needs of the storage admins and networking teams. It’s great that FCoE reduces the number of devices, amount of cabling and power/cooling requirements but at the same time it is bad that data paths are shared to the degree available as it can lead to bleed between the networking guys and the storage guys.

    I would expect that a lot of early implementations will still keep the storage and networking paths logically separated, i.e. a network card for TCP/IP traffic and a HBA for FCoE, with separate trunks/paths all the way through the infrastructure (probably using VLAN tagging). It’s the only way to guarantee to both networking and storage teams that their traffic has equal priority.

    I work with a relatively small setup (VMWare, blades and Netapp). I’m not a storage guru. I currently utilise both iSCSI and FC in my environment. FCoE would not change much for me, but I can see the datacenter taking a big advantage. The standard isn’t going to be the issue, it’s the management of converged traffic that will be the big one. It’s similar to when voice/video came onto TCP/IP, suddenly there was traffic that needed priority. Voice/Video is easier to manage as we know bandwidth requirements in advance. Storage is generally not so uniform. 10GB will quicly become 5G data 5G storage, or similar weighing. At least this way we can guarantee the throughput to each of the different teams.

    Gary

  6. slowe’s avatar

    Justin,

    I’ll grant you that FCoE has less overhead, there’s no doubt about that. But I really have to question the “compatibility” of FCoE with existing FCP–how does having to use new adapters (CNAs) and new switches (Cisco Nexus or equivalent) equal compatibility? Sure, it’s still FC, but you’ll need a bridge to join FCoE and FCP fabrics. I think that a lot of people overlook the fact that new CNAs and new fabric switches are required.

    As for performance, of course a 4Gb FC array via FCoE over 10Gb Ethernet will be better than a 1Gb iSCSI array over 10Gb Ethernet. The bottleneck here is the array, not the fabric or the transport.

    You’re betting FCoE for compatibility, but to me iSCSI seems much more compatible.

    Rodos,

    As I mentioned to Justin, it seems to me that we’ll need an FCoE-to-FCP bridge to join the physical fabrics into a single logical fabric. This bridge will introduce less latency and overhead than an iSCSI-to-FCP bridge, but it will be an additional piece of equipment that will be required. FCoE does seem much more of a transitional technology than anything else…hence my comment about VHS vs. Betamax. VHS was not the “best” technology, but it won. Will we see the same with FCoE and iSCSI?

  7. Roger Lund’s avatar

    slowe,

    I tend to agree with you, additionally, it is very easy to scale iSCSI, where you are stuck with one controller on a EMC / Netapp FC array, each iSCSI array ( EQL) has it’s own controller.

    Therefore, if you have three racks of EMC or Three Racks of EQL, ( 12 per rack ) the array each have two controller’s each ( or at least most that I have seen do ) where the , EQL iSCSI, would have something like 36 controllers, vs three EMC controllers for the same amount of storage.

    Now Even if you had 8 GB FC, wouldn’t you be limited to 8GB X 4ports X 3 Controllers = 96 GB

    To make it even, lets say you had iSCSI sans with 10 GB controllers, 10 GB X 2port X 36 Controllers = 720 GB

    Hence, if you had a Six top end switches @ 10 GB, Connected to 36 10 GB Sans all on the same switch back plane, wouldn’t the EQL have better throughput than FC Over Ethernet?

  8. Stu’s avatar

    Scott,
    Sure there is new equipment (CNA, CNS), but from a management standpoint, the new servers get added to the existing FC fabric and can be zoned and given access just like they were more FC nodes – this is the “easy interoperability”. There are plenty of customers running hundreds or thousands of nodes of FC. For customers that don’t have a large investment in FC, iSCSI is a good solution (and sure, it will be able to take advantage of 10GbE and CEE). iSCSI has done very well in the commercial/SMB space, but the ecosystem and tools for a large (hundreds of nodes) hasn’t developed yet. 2 paths to get customers to the 10GbE converged network.
    -Stu

  9. slowe’s avatar

    Roger,

    Thanks for your comment. Not all iSCSI arrays scale in exactly the same fashion, so some of what you are discussing may be specific to Dell/EQL. In addition, not all iSCSI initiators will scale overall throughput linearly with more links as well (think VMware ESX’s software initiator, for example). In this regard, I will say that FC (and presumably FCoE) have the advantage.

    Stu,

    “Easy interoperability” between FC and FCoE I will grant. As you describe, the ability to manage the FC and FCoE fabrics in much the same fashion, perhaps from the same tools, is quite compelling for large FC shops. But interoperability is not the same as compatibility, and to say that FC is compatible with FCoE is, in my opinion, incorrect. Perhaps I’m being a stickler, but if I can’t plug an FC HBA into an FCoE fabric then they’re not compatible. Interoperable, yes, but not compatible.

    Thanks to both of you for your comments!

  10. Nate’s avatar

    FCoE may be appealing to current FC users because they want interoperability, but I don’t see it having significant value over iSCSI for anyone incoming to the storage market. FCoE may use ethernet, but that doesn’t make it the easy plug into the network that iSCSI is. Particularly for SMBs and SMEs that may not have dedicated storage teams the ability to not need to learn an all new network is huge. A standard switch running standard configs is perfect for iSCSI, not so for FCoE. Routability is a big deal. When you want to be able to replicate data between data centers or even across a WAN link it’s nice to not have to take an extra step of tunneling or conversion. iSCSI maintains a leg up on cost as well because you don’t need special switches. FCoE may get you to the same number of switches as iSCSI, but not necessarily the same commodity level of switches. HBAs are also not needed in many scenarios. If your servers aren’t heavily loaded (which most aren’t) they can easily handle the little bit of extra work to run a software initiator. The Microsoft iSCSI initiator is fantastic with great MPIO. I’m biased because I was an early iSCSI adopter (started buying Equallogic back when they were just Equallogic), but I don’t see any value in FCoE other than giving FC clingers a platform from which to claim they are keeping up with the times. 10GB iSCSI would have meant the end of FC, so they had to jump the ethernet bandwagon.

  11. Roger Lund’s avatar

    slowe,

    Correct, and I think that really the largest bottle neck becomes the San and or Server / Servers.

    But it think that iSCSI is very flexible today, and will be more so in the future.

  12. Jose L. Medina’s avatar

    Scott:

    I agree with you: I cant see any reason to substitute iSCSI by FCoE. I think FCoE is another strategy to assure new bussiness to storage & networking vendors. Personally, I was using iSCSI for years in ESX environment without any special knowlegde of Pure Storage networking.. and it’s works good for me!. FCoE hide the manifest incapacity (or desire) of networking manufactures to improve ethernet with QoS capabilities (as an storage network requires). I’m sure iSCSI over serious datacenter ethernet can provide the same solutions of FCoE… without the expesive knowlegde & management of a FCxx network.

  13. Dan McConnell’s avatar

    Scott,
    Great question!… always fun cutting through the spin of the day to get through to reality. Appreciate the post and your thoughts/insights as they do cut through the spin cycle. Apologize up front for the length of the post…but getting caught up on much of the great discussion in the thread. So, on to the questions:

    1. FCoE is always mentioned hand-in-hand with 10 Gigabit Ethernet. Can’t iSCSI take advantage of 10 Gigabit Ethernet too?
    A->>In short… Yes. iSCSI will function on both non-DCB enabled as well as DCB enabled 10Gb Ethernet. For those that don’t need DCB or don’t want to invest/replace their infrastructure with DCB enabled switching, iSCSI will run just fine on “standard” 10Gb (or 1Gbps Ethernet for that matter unlike FCoE which requires 10Gbps Ethernet). For those that desire the DCB functionality, iSCSI will sit on top of a DCB enabled network and take full advantage of what DCB provides. (side note.. DCB-Data Center Bridging = CEE).

    2. FCoE is almost always mentioned in the same breath as “low latency” and “lossless operation”. Truth be told, it’s not FCoE that’s providing that functionality, it’s CEE (Converged Enhanced Ethernet). Does that mean that FCoE without CEE would suffer from the same “problems” as iSCSI?
    A->> DCB enabled networking (NICs, switches, and storage arrays) is required for FCoE. FCoE will not work without it. The reason for this is the fact that FCoE itself does not include a mechanism for ensuring reliable delivery. It therefore requires that functionality to exist in the network (ie flow control for Ethernet), which is what a DCB enabled network infrastructure is targeted to provide.
    iSCSI, on the other hand, has its own method for ensuring reliable transfer in the protocol layer (ie TCP). This enables iSCSI to run reliably on standard non-DCB enabled Ethernet switches (or remotely for that matter)

    3. If iSCSI was running on a CEE network, wouldn’t it exhibit predictable latencies and lossless operation like FCoE?
    A->>Yes

    Catching up on some of the interesting points/statements in the comments:
    – Justin mentioned some additional work required for iSCSI. This additional work(ie TCP) is what ensures reliable delivery in non-DCB enabled networks. FCoE pushes this work into the network and is why it requires DCB enabled NICs, switches, and storage devices. I would argue that for many typical workloads this additional processing is not noticeable. But in either case, if it is a pain point, iSCSI HBAs are available that offload this additional work. With the iSCSI HBA, the host side processing is equivalent to FC or FCoE (all enter under a similar storage stack).
    I guess one way of looking at it is as follows: Both FCoE and iSCSI can leverage optimized HBAs(DCB enabled FCoE CNAs or iSCSI offloaded HBAs) and DCB enabled switches to achieve similar performance… but iSCSI also has the flexibility to use standard NICs with standard non-DCB networks.

    – As far as Rodos’ point for fitting into existing FC frameworks. One question that comes to mind would be if those frameworks are integrating manageability for the Ethernet switches/networks? I would guess that both FCoE and iSCSI are in the same boat here.

    – Justin also brought up an interesting point that iSCSI is routable where FCoE won’t be. This does have some interesting implications today with routing’s ability to enable remote mirroring/DR. I would also suspect that it may become an even more interesting differentiator with the growth of cloud computing.
    I guess I’ll wind down with a tie to Nate’s point. FCoE might be appealing as a bridge back into existing Fibre Channel, but if the storage guys already have to swap out their network infrastructure toward ethernet, iSCSI’s flexibility to solve both ends of the cost/performance question and the fact that it is already here would seem to give it a leg up.

    -Dan

  14. Aneel’s avatar

    I’m not a storage guy either, at all. 100% data networking background.

    For q2: FCoE without CEE is a non-thing. Practically, just consider FCoE short for FCoCEE. Getting FC into the E frame and getting lossless, nonblocking, PFC, etc., capabilities into E were just steps in making FCo(C)E(E) a viable technology.

    And q3: As things stand today with the standards in progress, iSCSI would ride on the lower order priority queue in CEE and not get the same nonblocking, etc., that FCoE will. A software hack or specialized CNAs could change that, but none are being publicly discussed afaik.

  15. Jeff Asher’s avatar

    Let me start by saying that I am and have been for a long time an IP/ethernet proponent and regularly tell organizations not to invest in new FC infrastructure if they don’t already have one. It just doesn’t seem to make financial sense with the DataCenter Ethernet Initiatives in play at the moment. However…

    While the technology debates are fun, other aspects must be considered for this technology to be deployed and accepted. At most large organizations politics drives technology decisions at least as much as the merits of the technologies being considered. This is sad, but mostly true. The technologies being debated here actually intensify the political debate.

    Fibre Channel over Fibre Channel (FCoFC?) solves a political problem.
    FCoE creates a political problem.

    FC switches and infrastructure are popular not only because of many of the benefits and technical superiority in some cases over legacy Ethernet, but because the storage group often got to own the switches and infrastructure rather than the network group. One group owning the infrastructure from end-to-end had the benefit of own group being able to manage all aspects of storage without dependence on another group. Service levels could theoretically improve and political empire builders were happy because they owned more stuff and possibly more people. I’ve seen many 4Gbps FC deployments where iSCSI was more than adequate technically and the financial benefits were simply not debatable because the storage groups did not trust/like the network operations groups.

    FCoE throws a kink in things because the network operations groups are more likely to own the switches rather than the storage groups. This breaks the end-to-end model and theoretically would drive down service levels because of the interfaces required between the 2 operations groups (I actually believe service levels would increase in well run shops, but that is another debate).

    The problem is that while 10GB and DCE benefit both iSCSI and FCoE, they have the same political problems that have slowed the adoption of iSCSI in large enterprises. If the storage group doesn’t get to own the infrastructure from end-to-end, they are going to stick to FC regardless of the benefits of doing something else. And no, Role Based Access Controls for management doesn’t cut it in terms of the political problem.

    Is this view cynical? Probably, however it was developed from not just my own experience and those of many people I’ve talked to at many customers, various manufacturers, and resellers.

    Again, I say all this despite living clearly in the Ethernet camp.

  16. Nate’s avatar

    Jeff, that is a good point. I would venture also that the political reasoning hampering iSCSI and FCoE in the large enterprise is what makes the two technologies more appealing in the SMb and SME market. The smaller shops are less likely to have the luxury of dedicating teams of people to only storage, so they need crossover knowledge. I personally think iSCSI offers more accessible crossover knowledge due to the fact it can run on any network.

    The one way around the political issue for the larger folks is still to run a seperate physical network. Cost effective? No. Most efficient? No. Like you said in a well run shop the two teams working together should actually be better, but we all know in some cases they’ll still want to run their own gear. Basically at that point iSCSI and FCoE just become enablers of 10GB rather than convergance. That’s sort of ok though as I see it. When I first built out my iSCSI SAN I did so on the same standard Cisco switches I was using in the data network, but kept it physically seperate. I didn’t have political reason of course unless I wanted to be in a political battle with myself since I also work on the data network. I just knew the data network was peaked out and not ideal to handle the load. Now we are upgrading the infrastructure and bringing the SAN back into the mix. That’s the kind of flexibility I like about iSCSI.