Actually, based upon my most recent costing using 3 Dell M1000E, Infiband saves me 60K on deployment costs.

So right now it is much more cost effective versus Ethernet…

He went specifically at cost as a reason to put the ASIC at the far end of a “dumb” link, but in the real world the combination of “dumb” high speed ASIC plus “smart” ASIC in the switch is going to be more expensive than just a “smart” ASIC in the CNA because the CNA is going to be based on a platform that’s selling probably millions of units a year.

if you followed the discussion (I know it’s painful…. 68 comments with this one) you’ll notice that the discussion was revolving around “potential limitations” of the CNAs/FCoE when it comes to physically aggregate more secured network segments onto one cable vs using dedicated network cables.

The discussion was not so much “my kit is faster, cheaper, etc than yours”.

Massimo.

>I am not ruling out that there is lots of politics around this matter. So may be >the tactic of asking the right question may be leading to a positive technical >agreement but yet you may find other “obstacles” down the road.

Well there is a bit more to it than asking the right questions (customer education is absolutely key here) but you do need to make sure you direct your efforts to the right people in a given organization. i.e. don’t convince the systems guys that a given network IO technology is secure if they are beholden to a network security group for example.

>the module can support the hairpin mode that you’re talking about where a >packet that needs to be forwarded between two vNICs on the same port >would be sent out of the port, reflected back to the port by the external >switch, and then given to the destination vNIC. Exposing this capability is >just a matter of if and when products that can do something useful with >this become available, and users decide that they want to use that mode.

A large portion of my client base has to deal with PCI compliance. The demand is there. With whatever IO consolidation solution I use, I’m going to want to be able to mirror the traffic elsewhere, enforce private VLANs, and possibly use traffic based QoS.

>
>In addition to the intelligent adapters, the downstream switch that >recognizes the VN-tags and can appropriately handle these tags and route >traffic appropriately.

To be fair to Cisco, if you have UCS blades you already have this switching device (it doubles as the primary management device for all the subsequent blades) so you aren’t going to have to go buy anything new.

The UCS approach ( and likely the approach of the other FCoE solution vendors…i.e. Brocade, Qlogic, Emulex, HP, etc.) is the have a very intelligent CNA plugged into each server…..this devices is very complex because it includes a full SCSI state machine, FCP protocol engine, a full 10Gb Ethernet NIC, maybe even some TCP offload technology, some policy engines that can manage the multiplexing of the storage traffic and network traffic onto the DCE fabric, Pause-frame capabilities to participate in the fabric, etc. In the case of hte Cisco Palo adapter (per its marketing material and employee boasts) there are also some queueing structures to manage multiple MAC addresses, multiple WWNs, and the various driver instantiations suitable to present the multiple interfaces to the OS/Hypervisor….as well as the proprietary VN LInk Tagging to prevent loops. Each server in this model requires atleast one of these intelligent adapters.

In addition to the intelligent adapters, the downstream switch that recognizes the VN-tags and can appropriately handle these tags and route traffic appropriately. Cisco calls this End Host Mode…and efectively creates a static route between the virtual port traffic and an uplink port….a little similar to waht HP does in its Virtual Connect (but VC does not include the tagging part and doesnt support the multiple vNICs on a single physical port). I am sure Brad will correct me if my description is wrong…..as I said, i am just a Marketing guy.

Now, on the Virtual IO side (Xsigo, potentially Virtensys, Next IO, etc.), the system can have the same Intelligent CNA….with all of the same cool capabilities……the difference is tha tthe CNA lives in a remote appliance and connects back to a whole bunch of servers through an appropriate transport (Infiniband, PCI-Express, RDMAoverEthernet, etc.) In this model, the CNA with its 128 vNICs and vHBAs can now be shared across a large number of servers thus spreading the cost of this expensive resource. Since VMware can only support 32 Physical NICs, the additional 96 NICs available on the Palo chip may not be able to be used anyways.

In this distributed IO model, any server can attach over the server fabric and share the CNA resources. AND if the solution is designed properly, a single server can share the resources of multiple Adapters simultaneously. For example, suppose a customer has a legacy FC infrastructure and wants to move to FCoE…..or wnats to leverage their existing 1Gb switch infrastructure, but later move to 10GbE……they can easily deploy a 1GbE adapter in the Virtual IO appliance….and all servers can share this resource. LIkewise, when they want to move to 10GbE, they only need to add in a 10GbE adapter and all servers can share this resource. Should a server have the need for extreme performance, that server can now share multiple adapters….lets say 4x10GbE by creating 4 virtual NICs each terminating on a different Adapter.

So, you see, the big difference between the architectures are…..whether the intelligent adapter is a fixed resource that is physically tied to a server….or whether the adapter is a shared resource

Now, back to the 40 server example where each server connects to 10 different networks. For the Virtual IO solution, 1st, if the Intelligent adapter is designed appropriately, the adapter will allow for local switching….such as the new 64 lane Intel NIC. So, in the Virtual IO model, there can be 10 Palo CNAs in the IO Director. Each Palo CNA can create a vNIC that is assigned to each server for Network #1. On the 2nd Network adapter, 1 vNIC can be assigned to each server representing Network #2….etc. up to 10 Palo adapters. Each adapter will uplink to a different network. And if the Palo adapter were designed properly, it would do on chip switching with full bandwidth based QoS.

As you can see this model provides for significant flexibility to add in adapters based on network segmentation requirements, security requirements, bandwidht requirements, transport requirements, etc.

For those of you who remember back to the old Mainframe days, this is essentially the model that was deployed…..Channel IO from the Mainframe processor complex to the IO complex where an IO processor allowed for different IO adapters to be shared across the Mainframe processing resources. Now, since we have all been reinventing the mainframe architecture for the last 30 years…..THIS is the most obvious approach.

Regards

good stuff. I agree with most of what you say.

I am not sure about:

>This is mostly a matter of asking the right questions to network security
>people rather than making assumptions.

Customers that “have” to implement that high number of NICs recognize it’s not optimal. If it was that simple they would sit down with the networking people and would ask those question to get this fixed. After all it’s in their interest.

This is exactly what happened with the friction between Infrastructure people pushing VMs and Application people asking for physical dedicated servers. At the end of the day the Infrastr people pushed so hard (bottom up and top-down) that Application people had to accept that.

This is not happening for consolidating I/O (as far as I can see at least). Why is that? There must be more than asking the right questions.

I am not ruling out that there is lots of politics around this matter. So may be the tactic of asking the right question may be leading to a positive technical agreement but yet you may find other “obstacles” down the road.

Vendors also need to sell where politics rule…. so having the “best product” doesn’t always mean you get the deal done (if it isn’t aligned with the “politics”).

Massimo.

this won’t add anything to the topic but I really believe this has more to do with security than anything else. At least based on my feelings.

We have seen an exponential growth in capacity/performance for all subsystems in the server. This is true for CPU, disk, memory (at a lesser extent though) and certainly I/O is not an exception. If you look at the progresses in the FC and Ethernet realms… they have been exponentially growing compared to the actual usage and requirements (I think that most people on this thread would laugh at marketing statements such as “8Gb FC will double your 4Gb FC performance” etc etc).

My point is: if it was for performance (or management) reasons you could have solved the problem consolidating those 8/10/12 1Gbit Ethernet connections onto a couple of 10Gb Ethernet adapters. This is not happening and that’s for security reasons (IMO). We may discuss the “spikes” issue as you don’t want different traffics to step over to each others but:
a) the capacity/bandwidth/performance is so much that these situations are becoming niche and
b) there are rudimentary capabilities in the OS (certainly with ESX) to limit this effect (traffic shaping etc etc).

As I said I don’t think we need to discuss this further and further as it’s pretty much a different point of view based on different feedbacks/feelings.

I am not taking for granted I am right 100%!

This is a great discussion and I think we all learned lots of stuff. Thanks Scott for hosting this.

Massimo.

Currently, the vNIC I/O module is configured to handle the forwarding between vNICs that terminate on the same port because that’s the mode that makes sense considering what the external Ethernet devices such as switches expect at this point.

However, the module can support the hairpin mode that you’re talking about where a packet that needs to be forwarded between two vNICs on the same port would be sent out of the port, reflected back to the port by the external switch, and then given to the destination vNIC. Exposing this capability is just a matter of if and when products that can do something useful with this become available, and users decide that they want to use that mode.

Regarding VLANs vs separate networks – as I mentioned before, security is the issue only in some cases. In other cases, people use separate networks for performance isolation or for ease of management. There’s also the question of vSANs vs separate SANs which is relevant here.

Thanks for your comments,

Ariel