ESX Server, IP Storage, and Jumbo Frames

Maybe, if you can find the time, you can do some testing with iometer with jumbo frames enabled and disabled to see what the difference is performance wise… would be very interesting to know I guess.

and for Cisco, does changing the MTU size system wide affect the other attached devices in any way?

Hi Scott,
have you done also some benchmarking in order to determine if/how much throughput improvement exists on jumbo frame configuration?

thanks

Fantastic — I’m really hoping for official support for this soon (would love to drop the # of interrupts my servers spend processing storage traffic….not enough to get an iSCSI HBA though ;-).

Does anyone run any performance tests against iSCSI and NFS data stores with and without jumbo frames enabled?
We run tests against NFS data store (NetApp FAS3020) and didn’t see any significant performance improvements. We tested random read/write I/O workload using iozone with 8k blocks, but didn’t test any sequential loads.

Dmitry

We are looking into using Jumbo frames on our 6070 and Sybase/SUN5200 over NFS. We have Cisco 6509’s…

Do you have any general performance numbers on using Jumbo frames on Netapp?

Hi,

We’ve been running Jumbo/NFS in production for nearly two months now. No issues so far

Hi,

Does anyone know if this also applies to HW ISCSI HBA’s also?

On the topic of performance with jumbo frames I had 2 machines setup and tried with and without, while I saw no real improvement in performance the thing that I could see was that the amount of hit on the processor of the ISCSI target dropped noticable. (can’t tell you specifics but was at least 10-15% easily.

I’ll be setting up a full blown ISCSI/ESX setup over the next few. If anyone thinks it’s worth checking again I’ll put some stats up.

Ray

I’ve got to add this one for the noobs (i.e. Me) to ESX 3.5 and iSCSI w/ jumbo frames. You will need to actually add the empty port group to the new vSwitch using the command below BEFORE creating the VMkernel interface (otherwise you’ll get weird errors about the port group not existing). This step is mentioned in the main post, but the command to actually do it isn’t.

esxcfg-vswitch -A vSwitch1

And, thanks for the info! We have jumbo frames currently running on both the VM data and iSCSI networks (EqualLogic arrays w/ 6509s) and haven’t noticed any issues yet.

OK, the full command was chopped off. Let’s try that again…

esxcfg-vswitch -A port_group_name vSwitch1

So if jumbo frames are not supported for iSCSI or NFS, then a nice compromise would be to use normal frames for the Service Console port and its switch, and set jumbo frames for everything else. Thus, skipping the esxcfg-vmknic command?

Hi

I though that the software iSCSI initiator running on the esx host did not support jumbo frames?

If I just want jumbo frames from within my guest OS (Windows 2003 R2), I whould only need to configure the MTU on the vSwitch right

(esxcfg-vswitch -m 9000 vSwitch1)

And no need to a vmkernel?

Thanks
Felix

Just need to be sure I got it right

So jumbo frames actually do work from the iSCSI initiator on the ESX?

Thanks

Hi

I can’t get the jumbo/mtu 9000 to work with the microsoft iscsi initator from my Windows 2003 ENT R2 SP2

It works with the ESX iSCSI initiator

Thanks for a great post

Hi Fekix,

You need to use the enhanced vmxnet driver that comes with ESX 3.5 to get jumbo frame support for guest operating sytems. You also get the new TSO TCP Offload feature which further reduces CPU utilization.

http://www.vmware.com/support/vi3/doc/whatsnew_esx35_vc25.html

Hi… I have our ESX server connected to our Dell PS5000 currently using standard frames, is there anyway to ‘reconfigure’ the exisiting vmkernel and vswitch, without having to create another ?

Thanks Slowe…. the only problem I am now having is -

esxcfg-vmknic -a -i 172.16.1.1 -n 255.255.0.0 -m 9000 IPStorage tells me that -m isnt a valid switch. Im doing this via the CLI on my pc and not the server directly. Could this be the issue ?

Yeah Remote CLI and i d/l the latest version yesterday and the same issue.. I am using 3.5.. What directory do I need to be in on the ESX server if i try and run these commands.. Logging on as Root and running them in the root directory doesnt work, cannot find command or the like… I really appreciate the help you are giving me here !

Never mind i found it… YAY for the linux nuffies… thanks again for the great help…

Got it working.
What is the command to add this VMKernal port group to the new vSwitch1 with mtu 9000 and VLAN 200.
I use 6 nics to each of my esx servers. 2 are port-channel trunked to vSwitch0 nic0, nic1. 4 nics are port-channel trunked to vSwitch1 which is used for my storage network.
I got everything working by deleting my current vmkernel port group. then using the command to add a empty port group. then using scotts command to add the new IPStorage port channel with mtu 9000. then i just used the GUI in VC to make the port group vlan 200.

Thanks
Jason

Per the instructions above, I configured jumbo frames on my celeros san and esx 3.5 update 2 using a pair of 10g chelsio nic, one in my san and one in esx.

WOW..

Without Jumbos enabled, my throughput was about 200 MB/s. I was questioning if I should continue using 10G. After enabling Jumbo frames, my thoughput went to over 400 MB/s writes and 500 MB/s reads..

I can’t believe the difference. I wish VMWare would officially support it.

Thanks
Steve,
sjrieks@syr.edu

Hi,
I try to run this Switch command on my ESX 3.5 update2 on HP BL25p servers.

esxcfg-vswitch -m 9000

givs the error:

Unable to set MTU to 9000 the following uplinks refused the MTU settings: vmnic3

Any suggestion of what kan be wrong??
The other commands works fine!

//Thomas

Sorry correct command I run is

esxcfg-vswitch -m 9000 vSwitch3

still the same error:
Unable to set MTU to 9000 the following uplinks refused the MTU settings: vmnic3

Thanks slowe!

Seams that it´s not much to do now
Iam using the NetXtreme BCM5703 Gigabit Ethernet in full 1Gb mode. I can see on the Internet more than me have the same problem. The card is supporting Jumbo Frames in Windows but not with ESX. I guess the limitations is in the network driver in ESX.

//Thomas

Hi Scott,

This info is valuable!
I am running into this:

When running vmkping -s 9000, the amount sent is actually 9008, and doesn’t pass through.
I was able to push through a command of vmkping -s 8954 (sending 8962 bytes) and that was successful.

And idea why that may be?

I am using a Cisco switch (mtu 9000) and a Dell (also set to mtu 9000), and my 3par machine is _definitely_ set to mtu 9000.

Thanks for this blog!

After enabling the JumboFrame support like you said, the command vmkping -s 9000 10.0.10.1 do not return value. But I can’t ping at 8991 becase ping seam to add a 8 bytes overhead. It is normal?

PING 10.0.10.1 (10.0.10.1): 8991 data bytes
8999 bytes from 10.0.10.1: icmp_seq=0 ttl=64 time=1.640 ms
8999 bytes from 10.0.10.1: icmp_seq=1 ttl=64 time=1.363 ms
8999 bytes from 10.0.10.1: icmp_seq=2 ttl=64 time=1.362 ms

Thanks

@Mike

I do have also the same problem with PowerConnect 6248 switches. vSwitch and vmnic return well a MTU of 9000. The Equallogic storage device has its mtu set to 9000 and log returns a successful initiator connection. I get all volume presented to my ESX machines but I am unable to create a storage. It times out, ESX complains it cannot find a valid path.

Thanks for the great info, I made the adjustments to my vSwitch and the ping works.

Is there any other action needed on the Virtual Machines, to actually use these settings, or are the forced by the vSwitch?

Is this supposed to work on ESXi too? I was able to change the MTU on the vSwitch, but calling esxcfg-nics -l shows that the two NICs assigned to that vSwitch are still using a value of 1500. In addition to that, running esxcfg-vmknic with the -m flag returns the error “Unknown option: m”.

Has anybody tried this on ESXi? We have a normal ESX 3.5 license, but I don’t want to install and activate the full-blown until we’ve tested it carefully, so we were testing ESX 3i on a test server (same hardware as the production one, though).

Thanks,

Daniel

Hey Scott,

Very useful post!

Unforunately I’m running into the same problem as Pierre and Mike. When MTU is set to 9000 on the vmknic, you can only transmit 8991.

Any suggestions would be greatly appreciated.

Scott,

Thanks for the reply. According to Intel, the NICs I’m using should support jumbo frames; the machine has an Intel quad-port PCIe GbE controller (and two ports integrated in the motherboard, but I wasn’t using those). For some strange reason, VMware is using the e1000 driver for the NICs on the PCIe card (82571EB) and the igb driver for the ones on the motherboard (82575EB).

At any rate, even if I did get ESXi to change the MTU on the NICs, I would still have the problem of not being able to set the MTU on the VMkernel port.

I guess I’ll try using the integrated ports for the iSCSI connection to see if I can change the MTU on those, and if I still can’t change it for the VMkernel port I’ll try using ESX instead of ESXi.

OK, this gets weirder by the minute: the NICs I’m using for iSCSI connectivity are on vSwitch1, which I created specifically for that purpose according to the ESXi docs. Trying to change the MTU for that vSwitch doesn’t affect the NICs, but if I change it for vSwitch0 (currently being used for the management network and the RCLI VM), the change is applied to all of the NICs on the server, including the ones which are not assigned to any vSwitch!