Thinking Out Loud: Why Deploy FCoE?

Ordinarily I would agree with you Scott, but the storage portion of the register was actually another storage site, and was absorbed by the register in the last few months. The old site was on many a readers favorites list, as they had really interesting, and often (to that point) unknown data, which was always proven to be true. I’m not much for the register, but for the storage piece and of course BOFH.

That said, on topic….I’m pretty much not a fan of either fcoe or iscsi, FC gear is at an all time low, and is a ferrari compared to ip protocols. Fcoe isn’t mature yet (still has some issues, one of which you’ve pointed out)…If you’ve already got the investment, it makes sense to stay with native FC. 3 years will see a huge challenge, as the converged protocols become better, and then who knows, maybe we’ll all jump ship. But for now, there’s no compelling reason to move.

It does if your edge is blade chassis/servers. It costs about $20,000 to add fiber channel switches to a blade chassis. If I can get a blade chassis switch that offers a unified fabric at similar cost to current ethernet switches that can provide a significant cost savings for i/o for that blade system. Of course this requires CNA’s in the blades also.

I am tasked with architecting a NetApp based FCoE design and I’m trying to understand the benefits vs other topologies. If the design is focused on VMware, I see a lot of benefits around admin, capacity, provisioning, NDMP, etc. if I stay with an all NFS over standard 10GbE design that I give up when jump to FCoE. If the customer has no existing FC, I’m struggling with a good reason to go FCoE.

Today SRM doesn’t support NFS, but that should soon change. And also the FCoE adapters in the NetApp only support FCoE which means I would have to also include Ethernet adapters to accomplish CIFS, NFS, iSCSI if customer decided to take advantage of those protocols as well. But I think the intention is to have NAS protocols added to the FCoE only CNAs soon.

So in the future when the NetApp can do FCoE, NFS, CIFS, and iSCSI over a single adpater, and SRM is supported on NFS, my question is do the benfefits of running VMware over NFS on NetApp give the best features and functionality over any benefits that FCoE provide?

SLowe,

Only Cisco??? How about QLogic and IBM’s recent announcement?

http://finance.yahoo.com/news/QLogic-Single-Chip-Converged-pz-956357303.html?x=0&.v=4

Am I that transparent? Ok, yes I am. But I like Charles’ point/question if you don’t have FC what is the virtue of FCoE. To further extend that line of thought if FCoE is really built to transition away from FC wouldn’t it’s success deplete it’s own initial virtue? If FCoE really kills out FC then there will be no more FC customers to entice with the possibilities of getting on he 10gig bandwagon. At which point any new customers will be comparing a technologoy in FCoE that was designed to transition from a now dead platform vs a technology like iSCSI with virtues that hold true whether you are replacing an existing storage solution or brand new to the scene.

@Charles Taylor,

We’re dwindling out the last of our fiber-channel, making an all-NFS-over-10GbE push, and as of the end of last week, NetApp only had a first generation FCoE expansion card, and they had found flaws in it, and AT THIS TIME, do not offer an FCoE expansion card until they get the kinks worked out.

Be wary of that.

I’ve been running VMware on 1GbE NICs for over a year now, and with our recent inception of the Nexus 5020’s and 10GbE, it’s only going to get better. I mean, you can get Intel 10GbE-NICs for $600! Dual-port! That’s as cheap as most 2 & 4 port Intel PRO1000 adapters!

Ditch FCoE, guys. It’s WAY behind it’s time. The idea of Cisco unifying the fabric is a great idea that should have happened about 4-5 years ago, and we love the Nexus line, but I’ll agree with everyone else here: Another Cisco ploy to steer the industry into overpriced hardware.

@Scott,

I won’t redo the “Cisco is not the only player behind this” argument. You know where I stand on that.

All I’ve said is that the access-layer consolidation biz case will be good enough for enough customers to keep FCoE going. I still stand by that. Not the least because I’ve seen the customers. And I didn’t have to convince them. Neither did Cisco or Brocade.

For some subset of big customers with big $, the ultimate FCoE use case with differentiated CEE transport for all storage and data traffic presents a big enough net cost savings opportunity [devices, cabling, headcount, maintenance, firmware, spare part depots, etc] that they’ll give it a shot. And that subset is big enough and willing to spend enough $ to get there–that everyone else doesn’t matter.

Ignoring details and performance measurements: everything that you can do with FCoE you could do with competing or alternative technologies. And vice versa. Since when have the technical capabilities of a given technology been the sole determiner of their success?

Sorry - one typo (in a long, annoying comment

…This isn’t a secret. Look at the Netapp TR-3428 (and upcoming TR-3749) and EMC H6337 docs which spell the timeouts for NFS datastores on FAS and Celerra platforms respectively - which are in many tens of **seconds** (not minutes in my original comment), and for iSCSI, the recommendation is 60 seconds.

The current recommendation translates to 125 seconds, or 2 minutes 5 seconds, though you should refer to the latest docs for the best practices.

Scott,

I would consider Cisco as the only one who offers blades with connectivity to unified fabric if they were actually shipping product.

vina mentioned the IBM announcement and I can tell you that within six months there will be others with it also.

Heck, last year I was considering Infiniband for similar reasons but it seems it can’t gain any acceptance outside of HPC environments.

I’d say the original article that Scott references spells out why the convergence argument just isn’t good enough. It really isn’t convergence if it is just a gateway to a still seperate FC network that you have to maintain. If anything doesn’t that make things more complex to manage, not easier? To do FCoE we are still going to be talking about specialized storage switches. Yes it’s a network switch and a storage switch (well half a storage switch) slammed into one unholy case, but it’s not like you can just use any network switch. You’ll still need storage networking experts rather than being able to accomplish the task with your standard network guys and gals. If you run an IP based storage network the same knowledge set that works for your closet switches can apply to your datacenter switches. If you go FCoE you’ll need to know two differet platforms (three really because you’ll still need to know the pure FC too). I still really only see storage network manufacturers pushing the FCoE, not standard network manufacturers. I believe strongly that FCoE is an attempt to keep the need for a specialized piece of hardware in the datacenter that can be charged for at a premium. IP based storage solutions can be run on any switch giving the customer far greater choice and a better price point. While there may be some minimal value for current FC customers to have a way to get some convergence on the last mile, it’s not really helping them all that much. They’d be better served by simply improved FC, or biting the bullet and moving on.

Scott:

These seems to be a tendency to treat FCoE implementation as one monolithic event that should be depolyed edge-to-core for all customers in one fell swoop, which is not even close to the position Cisco has taken.

We have maintained that at that and edge-in approach makes sense. Consolidation in the server access layer help customers reduce TCO (fewer interfaces, cables, up-stream switch ports) and improve functionality (pervasive SAN access, better flexibility to support VMotion). With FCoE, we help customers accomplish this without disrupting the rest of the LAN and SAN environment (design, architecture, SOP), which is not something you can say if you were to try and move folks from FC to iSCSI to gain the benefits of a access layer unified fabric.

We believe propagation into will happen SAN, but it will progress at a more measured rate–a rate that makes business and technical sense for customers. The introduction of multi-protocol storage further supports this perspective.

For mid-market customers who don’t have an existing FC environment, FCoE simply becomes another option along with iSCSI–they are both storage over Ethernet solutions. I think the driver in this space will be more mundane things like which approach has better price points, which one is easier to set-up and support, and what is supported out of the box by MS, etc.

Anyway, the point is that no one is losing any options here–on one, well at least Cisco, is not saying protocols are going away or should go away–I said on a number of occasions that FC, FCoE and iSCSI will happily continue to coexist for number of years to come. Again, the emergence of multi-protocol storage would seem to support that view.

Finally, I know the whole standards thing is a favorite refuge of naysayers, but, the truth is that one is slowly falling apart. The FCoE spec is baked, and we are quickly closing in on the underlying Ethernet standards, to the point that the University of New Hampshire InterOperabilty Lab had a very successful DCB PlugFest back in May. I did a blog post about this a few weeks ago http://tr.im/nu71

Anyway, good post–love the discussion.

Regards,

Omar Sultan
Cisco

You miss the point.

Multi-hop lossless/FCoE would mean big, expensive, lossless/FCoE aggregation and core Ethernet switches to replace existing aggregation and core Ethernet switches and core SAN directors. While it might be an interesting proposal for a green field opportunity, no customer would rip and replace their entire Ethernet network just to get the advantage of consolidated I/O.

Second, if you are consolidating networks, the largest number of network ports are at the access edge, so the biggest ROI and shortest payback will always come by consolidating the edge. Think about it, I could upgrade my disk storage targets from 4Gb FC to 8Gb FC and cut my disk target ports in half. But I probably have as few as 4 or perhaps as many as 32 disk storage target ports in a typical SAN. I could upgrade interswitch link ports from 4Gb to 8Gb and cut my ISL ports in half, but that is only a small majority of my total ports. But I could have 500 or even 1000 host ports. If I could consolidate them, along with 4, 6, or even 8 GigE ports onto a pair of 10GigE consolidated FCoE ports, I get a significant reduction. If I have 500 servers, each with two FC ports, and four GigE ports, I can eliminate 2,000 ports from my network. That is huge.

Think of what this means in a blade server environment. Today, with separate GigE and FC a 14 to 16 blade server chassis might have anywhere from four (two Ethernet and two FC) chassis switches up to as many as eight (six Ethernet and two FC) chassis switches. Customers could cut that down to two FCoE switches and cut tens of thousands of dollars per chassis cost out of their platform.

What matters is, FC is multi-hop., and Ethernet is multi-hop. So I split my traffic out at the edge, and my FC traffic can cross as many FC switches as needed, and my Ethernet traffic an do the same.

Eventually, one data center class lossless Ethernet network sounds great, but it will take significant changes in how networks are managed, as traditional Ethernet network managers are IP centric, and lossless, routed at Layer 2 networks are new to them.

Nate:

I think it is a rare customer that will happily rip out perfectly functioning FC-attached storage to replace it with FCoE attached storage that does the same thing–I certainly don’t believe the storage vendors are building forecasts on this assumption. Therefore, any strategy needs to have an investment protection angle–how will you address the server-side challenges without disrupting a SAN that is quite probably happily chugging along.

So, yes, the port consolidation piece is a significant cost for whomever is on the hook for paying for server chassis, NICs, HBAs, cabling and access layer ports, so being able to fold multiple GbE NICs and an HBA into a single 10GbE link has a certain appeal to it and some compelling economics behind it for those paying that portion of the bill.

I am not sure why you consider FCoE a “complication”–from the infrastructure perspective it simplifies infrastructure and from an operations and management perspective it is managed exactly as before–minimal product training, same tool, processes, etc.

Omar Sultan
Cisco

Omar, I consider it a complication because it is an extra layer and an extra piece of technology just to handle the convergence/divergence for the last mile. You are still going to need both ethernet and fc networking gear on the backend of your FCoE splitter. I really don’t see a nexus as the same tool as a 3750 for example, and introducing FCoE will certainly mean new processes.

I’m still not buying the port consolidation even works out to be so great. First if you are thinking you are getting more bandwidth on the server side or are going to be able to live more ports due to FCoE, you will still need to add FC ports on the SAN side or you are just creating a bottle neck. Second scenario, lets consider a current FC customer that wants more bandwidth to the host both for storage and network so they can increase virtualization load up. Now your argument is that FCoE is to reduce the server port count, but if this customer is going to want 10 GB for network and let’s say they use an 8GB FC link (could be 16GB soon). So the server has two ports and two cables, one network one data (they could go 4 links for redundancy, but I’ trying to keep the scenario simple). Where is the benefit for FCoE here. With FCoE the customer would need two FCoE 10 GB ports to get the same bandwidth. SO there are still two cards and two cables, plus the customer must now insert a nexus switch to split the FC from the IP and get it sent to the respective switches that the customer must also still maintain.

I guess my point is there are three potential storage customers. Customer A has no current storage solution, Customer B has an FC storage solution and is willing to make some changes to get to consolidation, Customer C has FC and wants to make minimal changes and merely wants to maintain their network and keep up with the speed of business (I’m leaving out Customer D, the guy who already has an IP storage solution and sees no point in FCoE). What value does FCoE bring to customer A? I don’t see a single value brought to the table. To go FCoE they would need to buy all the gear to go FC PLUS the FCoE layer. Why not just straight FC, or even better IP based storage. If customer B is really looking for consolidation and is willing to make change to do so, hello IP based storage. Why graft a whole new layer on just to consolidate one piece of the puzzle. Customer C doesn’t really want change. It seems to me this is the real target of FCoE, but as in my above scenario I still don’t see the value. If you are in love with your FC and not wanting to make any changes why change to add FCoE to the mix. Just get the latest and greatest FC and keep humming along on your mary way. Your going to need to buy the latest and greatest FC gear for your SAN side anyways, otherwise it will fall behind the ethernet side sitting over in front of the nexus at which point it will be like your packes are cruising on a nice new highway then hit road construction.