Comments on: ESX Server, IP Storage, and Jumbo Frames

By: Olufisayo

Olufisayo — Fri, 20 Jan 2012 12:23:50 +0000

This is pretty cool

By: slowe

slowe — Fri, 14 Jan 2011 01:36:36 +0000

Thanks Ken—that’s useful!

By: Ken

Ken — Thu, 13 Jan 2011 21:08:35 +0000

Here is a nice single command line for ESX 4. This assumes the initial vswitch and vkernel port are already there and that the physical switch and SAN are already configured.:

esxcfg-vswitch –m 9000 vswitch0 && esxcfg-vmknic -d -p “IP Storage” && esxcfg-vmknic -a -i 192.168.1.13 -n 255.255.255.0 -m 9000 “IP Storage” && vmkping -s 9000 192.168.1.31

Obviously use the correct vswitch name, vkernel port name, MTU size, and IP addresses.

This runs VERY fast and can be done live. I ran it live on an 8 node cluster and had no VMs crash.

By: Chris

Chris — Thu, 09 Dec 2010 21:07:43 +0000

@Frederic
Thank you! I had found the -d flags meaning but went through the same thing…raising the packet size to 8972, after which it breaks. I thought it was not working because of this but I was not adding in the overhead – this makes perfect sense and thrills me to see that our setup (esx4.1/10GB => Cisco Nexus 7K => NetApp FAS 10GB) Is working brilliantly with Jumbo frames.

The killer for me was creating that vmk manually and then migrating it to the vDS…Scott you saved my life once again.

By: Fréderic Kinnaer

Fréderic Kinnaer — Thu, 11 Nov 2010 17:29:52 +0000

Great article, only one thing to note: when doing the test, vmkping -s 9000, it will always respond (try it with -s 12000); because the packets get fragmented if size > MTU (incorporating the overhead, even 9000 is too big, as its 9000b + overhead that gets sent over the link).
To really test, you should set the Don’t Fragemnt bit on the packets with the -d flag, then substracting the overhead from the 9000 bytes:

vmkping -d s 8972 172.16.1.200

I don’t know by heart the overhead, but found experimentally its 28b.

Greets

By: Shannon

Shannon — Tue, 18 May 2010 20:08:00 +0000

I know this is an old thread but to clarify for anyone that finds it, mtu size is seen from the host and switch side different. From the host when you’re setting mtu think of it as the data you’re sending, the payload. From the switch side it is the entire packet, the payload and the all the necessary wrapper. On enterprise Cisco gear the default, if you will, size for a jumbo packet is 9216. Most people from a host perspective, as is mentioned above, set jumbo packets at 9000. This gives 216 bytes of overhead that can be added without causing issues. The important thing to remember is to leave room or the network overhead, you can’t set your host and switch to the exact same number — it won’t work, you won’t be leaving room for the wrapper around the payload.

By: slowe

slowe — Mon, 12 Apr 2010 12:15:45 +0000

Scott Owens,

It sounds like you have a good grasp of link aggregation. Many people don’t, hence my comment. Carry on, good sir!

By: scott owens

scott owens — Mon, 12 Apr 2010 12:10:33 +0000

Thanks,

Yes, the lacp teaming characteristics are understood to limit network conversations to the maximum throughput provided by a single link; unlike routers the ESX load balancing occurs on a per stream basis instead of per packet. It is partially for this reason our new ESX servers get multiple 10 Gb ports to Nexus gear; the only way we can get more than 1Gb conversations is from 10 Gig nics – there are no 2.5 Gb or 5 Gb nics and sometimes 1Gb just is not enough ( backups?). We consider lacp teaming part of “standard build” in the same way we do speed and duplex settings – if we have a link down event we would rather lose an active link if there is a second interface already passing traffic versus having an inactive second link become active.

As you pointed out in the third bullet under the link aggregated examples -
“Traffic that is primarily point-to-point won’t see any major benefit from this configuration. A single VM being accessed by another single client won’t see a traffic boost other than that possibly gained by the placement of other traffic onto other uplinks.”
We agree with and design around that last sentence. And that was the reasoning for my LACP teaming comment in my last sentence.

By: slowe

slowe — Sun, 11 Apr 2010 19:56:02 +0000

Scott Owens,

It’s important to understand that link aggregation protocols like LACP will not—in the VMware context, at least—improve performance for a single datastore. Remember that you must have multiple source-destination pairs (multiple VMkernel ports and/or multiple IP targets) in order to take advantage of link aggregation protocols, and even then traffic to/from a single datastore will only be able to consume the bandwidth of a single member of the link aggregate (i.e., 1Gbps in a group of Gigabit Ethernet links aggregated together using LACP). I wrote a number of articles about this behavior when describing how VMware ESX/ESXi utilizes multiple NICs. This article is one such example:

http://blog.scottlowe.org/2008/07/16/understanding-nic-utilization-in-vmware-esx/

Hope this helps!

By: scott owens

scott owens — Sun, 11 Apr 2010 11:37:51 +0000

Mr. Werntz,
That is an interesting document. NetApp also has one on performance of Microsoft Exchange w/multi-protocols.

I have one complaint that seems to be a glaring fault.
The Fiber Channel interfaces tested are 4Gb FC, the IP protocols NICs are 1 Gb.
Not only is there a greater possible bandwidth via 4Gb FC if you read the equipment used the 4Gb FC HBAs are intelligent and would cost around $1000 to $1500. The 1 Gb NICs are commodity and run around $250 to $400 (some tests used quad nics ).

Hardware accelerated 4 Gb FC vs 1Gb IP on generic hardware. …..
And this is a valid test with results that surprise ?

How about 4 Gb FC vs 10 Gb NICs with TOE like the Chelsio or Broadcom iSCSI adapters ?
Here is a research document done at a couple of educational institutions reprinted at Chelsios website http://www.chelsio.com/hotinterconnect.html
Performance increases, CPU decreases, latency decreases. ..
I would suspect NetApp tests would show similar or better results especially with decent QOS and LACP teaming.