*) We need to move to 10GB/s ethernet soon
*) We have maxed out the number of expansion cards
*) Due to above I have to make compromises between backup (FAS-to-Tape) or multipathing diskloops.

I was hoping that FCoE would leviate our dilemas and maybe help us get a free slot for future expansion (maybe PAM card).

Now, talking to a Netapp engineer a month agou he told me that FCoE card didn’t support the “normal” ethernet trafic so that became it a showstopper as it makes little sence compared to exchanging our 4×1GB card for 2×10GB cards. Also, I got no answer if that is an implementation (software) issue or a card issue. At the same time as Netapps current 10GB card has TOE issues I’m realy not happy.

That said I see a possible future for FCoE in our show, but the nexus switches will be at the edge/parallel to our current core 6509’s until the become obsolete.

Thanks for the clarification. Everybody works for somebody, but I’m really only worried about affiliations when someone has a clear agenda but isn’t being equally clear about their affiliations.

I’m still not buying the port consolidation even works out to be so great. First if you are thinking you are getting more bandwidth on the server side or are going to be able to live more ports due to FCoE, you will still need to add FC ports on the SAN side or you are just creating a bottle neck. Second scenario, lets consider a current FC customer that wants more bandwidth to the host both for storage and network so they can increase virtualization load up. Now your argument is that FCoE is to reduce the server port count, but if this customer is going to want 10 GB for network and let’s say they use an 8GB FC link (could be 16GB soon). So the server has two ports and two cables, one network one data (they could go 4 links for redundancy, but I’ trying to keep the scenario simple). Where is the benefit for FCoE here. With FCoE the customer would need two FCoE 10 GB ports to get the same bandwidth. SO there are still two cards and two cables, plus the customer must now insert a nexus switch to split the FC from the IP and get it sent to the respective switches that the customer must also still maintain.

I guess my point is there are three potential storage customers. Customer A has no current storage solution, Customer B has an FC storage solution and is willing to make some changes to get to consolidation, Customer C has FC and wants to make minimal changes and merely wants to maintain their network and keep up with the speed of business (I’m leaving out Customer D, the guy who already has an IP storage solution and sees no point in FCoE). What value does FCoE bring to customer A? I don’t see a single value brought to the table. To go FCoE they would need to buy all the gear to go FC PLUS the FCoE layer. Why not just straight FC, or even better IP based storage. If customer B is really looking for consolidation and is willing to make change to do so, hello IP based storage. Why graft a whole new layer on just to consolidate one piece of the puzzle. Customer C doesn’t really want change. It seems to me this is the real target of FCoE, but as in my above scenario I still don’t see the value. If you are in love with your FC and not wanting to make any changes why change to add FCoE to the mix. Just get the latest and greatest FC and keep humming along on your mary way. Your going to need to buy the latest and greatest FC gear for your SAN side anyways, otherwise it will fall behind the ethernet side sitting over in front of the nexus at which point it will be like your packes are cruising on a nice new highway then hit road construction.

There appears to be quite the interesting discussion going on over at Scott Lowe’s blog regarding FCoE (Fibre Channel over Ethernet) as its relationship to the data center, VMware, and, pretty much everything else. …

I think it is a rare customer that will happily rip out perfectly functioning FC-attached storage to replace it with FCoE attached storage that does the same thing–I certainly don’t believe the storage vendors are building forecasts on this assumption. Therefore, any strategy needs to have an investment protection angle–how will you address the server-side challenges without disrupting a SAN that is quite probably happily chugging along.

So, yes, the port consolidation piece is a significant cost for whomever is on the hook for paying for server chassis, NICs, HBAs, cabling and access layer ports, so being able to fold multiple GbE NICs and an HBA into a single 10GbE link has a certain appeal to it and some compelling economics behind it for those paying that portion of the bill.

I am not sure why you consider FCoE a “complication”–from the infrastructure perspective it simplifies infrastructure and from an operations and management perspective it is managed exactly as before–minimal product training, same tool, processes, etc.

Omar Sultan
Cisco

Multi-hop lossless/FCoE would mean big, expensive, lossless/FCoE aggregation and core Ethernet switches to replace existing aggregation and core Ethernet switches and core SAN directors. While it might be an interesting proposal for a green field opportunity, no customer would rip and replace their entire Ethernet network just to get the advantage of consolidated I/O.

Second, if you are consolidating networks, the largest number of network ports are at the access edge, so the biggest ROI and shortest payback will always come by consolidating the edge. Think about it, I could upgrade my disk storage targets from 4Gb FC to 8Gb FC and cut my disk target ports in half. But I probably have as few as 4 or perhaps as many as 32 disk storage target ports in a typical SAN. I could upgrade interswitch link ports from 4Gb to 8Gb and cut my ISL ports in half, but that is only a small majority of my total ports. But I could have 500 or even 1000 host ports. If I could consolidate them, along with 4, 6, or even 8 GigE ports onto a pair of 10GigE consolidated FCoE ports, I get a significant reduction. If I have 500 servers, each with two FC ports, and four GigE ports, I can eliminate 2,000 ports from my network. That is huge.

Think of what this means in a blade server environment. Today, with separate GigE and FC a 14 to 16 blade server chassis might have anywhere from four (two Ethernet and two FC) chassis switches up to as many as eight (six Ethernet and two FC) chassis switches. Customers could cut that down to two FCoE switches and cut tens of thousands of dollars per chassis cost out of their platform.

What matters is, FC is multi-hop., and Ethernet is multi-hop. So I split my traffic out at the edge, and my FC traffic can cross as many FC switches as needed, and my Ethernet traffic an do the same.

Eventually, one data center class lossless Ethernet network sounds great, but it will take significant changes in how networks are managed, as traditional Ethernet network managers are IP centric, and lossless, routed at Layer 2 networks are new to them.

http://nohype.tumblr.com/post/111557682/emcworld09-slides

These seems to be a tendency to treat FCoE implementation as one monolithic event that should be depolyed edge-to-core for all customers in one fell swoop, which is not even close to the position Cisco has taken.

We have maintained that at that and edge-in approach makes sense. Consolidation in the server access layer help customers reduce TCO (fewer interfaces, cables, up-stream switch ports) and improve functionality (pervasive SAN access, better flexibility to support VMotion). With FCoE, we help customers accomplish this without disrupting the rest of the LAN and SAN environment (design, architecture, SOP), which is not something you can say if you were to try and move folks from FC to iSCSI to gain the benefits of a access layer unified fabric.

We believe propagation into will happen SAN, but it will progress at a more measured rate–a rate that makes business and technical sense for customers. The introduction of multi-protocol storage further supports this perspective.

For mid-market customers who don’t have an existing FC environment, FCoE simply becomes another option along with iSCSI–they are both storage over Ethernet solutions. I think the driver in this space will be more mundane things like which approach has better price points, which one is easier to set-up and support, and what is supported out of the box by MS, etc.

Anyway, the point is that no one is losing any options here–on one, well at least Cisco, is not saying protocols are going away or should go away–I said on a number of occasions that FC, FCoE and iSCSI will happily continue to coexist for number of years to come. Again, the emergence of multi-protocol storage would seem to support that view.

Finally, I know the whole standards thing is a favorite refuge of naysayers, but, the truth is that one is slowly falling apart. The FCoE spec is baked, and we are quickly closing in on the underlying Ethernet standards, to the point that the University of New Hampshire InterOperabilty Lab had a very successful DCB PlugFest back in May. I did a blog post about this a few weeks ago http://tr.im/nu71

Anyway, good post–love the discussion.

Regards,

Omar Sultan
Cisco