EMC Celerra Optimizations for VMware on NFS

Recently Jason Boche posted some storage performance numbers from his EMC Celerra NS-120. Out of those storage performance tests, Jason noted that the NFS performance of the NS-120 seemed a bit off, so he contacted EMC’s vSpecialist team (also known as “Chad’s Army”, but I like vSpecialists better) to see if there was something that could or should be done to improve NFS performance on the NS-120. After collaborating internally, one of our team members (a great guy named Kevin Zawodzinski) responded to Jason with some suggestions. I wanted to reproduce those suggestions here for everyone’s benefit.

Note that some of these recommendations are already found in the multi-vendor NFS post available on here on Chad’s site as well as here on Vaughn Stewart’s site.

In addition, most if not all of these recommendations are also found in the VMware on Celerra best practices document available from EMC’s web site here.

Without further ado, then…

  • As has been stated on multiple occasions and by multiple people, be sure that virtual machine disk/application partitions have been properly aligned. We recommend a 1MB boundary. Note that Windows Server 2008 aligns at a 1MB boundary automatically.
  • Use a block size of 8KB unless other recommended or required by the application vendor. Note that the default NTFS block size is 4KB. (Pages 128 through 138 of the Celerra best practices document contain more information on this bullet as well as the previous bullet.)
  • Turn on the uncached write mechanism for NFS file systems used as VMware datastores. This can have a significant performance improvement for VMDKs on NFS but isn’t the default setting. From the Control Station, you can use this command to turn on the uncached write mechanism:
    server_mount <data mover name> -option <options>,uncached <file system name> <mount point>
    Be sure to review pages 99 through 101 of the VMware on Celerra best practices document for more information on the uncached write mechanism and any considerations for its use.
  • Change the VMware ESX settings NFS.SendBufferSize and NFS.ReadBufferSize to a value that is a multiple of 32. The recommended value is 64. See page 73 of the best practices document for more details.
  • If you’ve adjusted the NFS.MaxVolumes parameter in order to have access to more than 8 NFS datastores, you should also adjust Net.TcpIpHeapSize and Net.TcpIpHeapMax parameters. The increase should be proportional; if you increase the maximum volumes to 32 (a common configuration), then you should increase the other parameters by a factor of 4 as well. Page 73 of the best practices document covers this. This VMware KB article and this VMware KB article also have more information.
  • Although not directly related to performance, best practices call for setting NFS.HeartbeatFrequency (or NFS.HeartbeatDelta in VMware vSphere) to 12, NFS.HeartbeatTimeout to 5, and NFS.HeartbeatMaxFailures to 10.
  • Ensure that the LUNs backing the NFS file systems are allocated to the clar_r5_performance pool. This configuration will balance the load across the SPs, LUNs, etc., and help improve performance.

Depending upon the other workloads on the system, another NFS performance optimization is to ensure that the maximum amount of write cache on the SPs is configured. However, be aware this may impact other workloads on the array.

As Jason noted in his post, implementing these changes—especially the uncached write mechanism—offered performance benefits for NFS workloads.

Keep these configuration recommendations in mind when setting up your EMC Celerra for VMware on NFS.

Tags: , , , , ,

  1. Jason Boche’s avatar

    Thank you for sharing. I hope this article was of the “sharing the information” type and not “urgent damage control”, making Scott work on the weekend. I’m very pleased with the EMC Celerra. EMC has a great team of storage and virtualization specialists (including yourself), which I am fortunate to work with.

    Reading all manuals/whitepapers, professional serivces implementation, and formal training would all go a long way in knowing how to best set up Celerra arrays. However, short of standing up a V-Max, I also believe in simplicity and hitting the ground running with simple to manage arrays. EMC’s Celerra Manager goes a long way in making Celerra management and implementation easy on the storage administrator. Many tasks can be completed with an easy to follow Wizard. Building in Wizard driven best practice tuning for VMware (and other platforms) would seem to be a good fit in this theme. Or at least in this case when creating NFS exports, a little checkbox asking “Is this for ESX?” could automatically tune the data movers enabling the uncached option.

    If you could manage FC with Celerra Manager (instead of NaviSphere), well that would just be too cool for words – single pane of glass management :)

    Thanks again!

    Jas

  2. slowe’s avatar

    Jason,

    This is definitely a “sharing the information” post—no “urgent damage control” going on here. Thanks for being willing to work with us to bring this information to light!

  3. Ian Stewart’s avatar

    A word of caution before turning on the uncached write mechanism for NFS – if your NAS code is between 5.6.43.8 and 5.6.45.5 you will need to upgrade (current version is 5.6.47.11) before enabling the uncached option.

    For more details, see the EMC knowledgebase article emc216760. You can view this by going to the Powerlink website and entering “emc216760″ under Search Support.

  4. PiroNet’s avatar

    There is a small typo Scott, it is NET.TcpIpHeapSize and NET.TcpIpHeapMax parameters…

  5. Yossi Mesika’s avatar

    Great post Scott.

    The VMware with Celerra best practice document includes further details on the first two recommendations listed here regarding virtual machines alignment with Celerra storage. Please review pages 128 through 138 for the section that covers this topic.

    It would be great if you can add this reference in the post itself.

  6. slowe’s avatar

    PiroNet,

    Pesky little typos—I’ve fixed those. Thanks!

    Yossi,

    I’ve added a reference per your request. Thanks!

    Ian,

    Thanks for the additional information!

  7. Derek’s avatar

    Does the 8K NTFS block size also apply to traditional VMDKs hosted on SAN Fibre Channel storage? Or is that setting unique to NFS hosted VMDKs?

  8. slowe’s avatar

    Derek,

    The EMC CLARiiON-VMware integration white paper also recommends an 8K allocation unit size unless the application or vendor recommends otherwise. See page 27 of this document:

    http://www.emc.com/collateral/hardware/white-papers/h1416-emc-clariion-intgtn-vmware-wp.pdf

    I hope this helps!

  9. ron’s avatar

    Scott,

    Any chance you know of any docs for using ESX 3.5 (or vSphere) and an AX-4? For direct attached storage, or is it just dead simple :)

    We just got an NS120 (not even powered on yet) and no one feels comfortable running VM’s off of NFS, and I cant sway them, so we’re also on the cusp of getting the AX-4 probably, so if you know of anything I appreciate it.

    Thanks very much

    Ron

  10. ron’s avatar

    Belay that capt. i think I found what I needed, unless you know of something better, http://www.emc.com/collateral/hardware/white-papers/h1416-emc-clariion-intgtn-vmware-wp.pdf

    Tx

    Ron

  11. karl’s avatar

    Hi,

    do you think that these Celerra Optimizations will work with XenServer (instead of ESX) ?

    Thanks

  12. Matt Mancini’s avatar

    We are just starting down the path of comparing our Celerra to NetApp and this was some helpful information to get us started out!

    thanks..

  13. Michael’s avatar

    I have done considerable research on this and I have concluded that there is no performance benefit from disk alignment in the GOS if the underlying storage is NFS (block based yes). This is because NFS separates the block based storage from the GOS and exposes a single disk to the VM. There is no performance benefit from aligning a single disk volume.

    That being said, you should still align the GOS as it makes the VM vMotion “compatible” with block-based storage without degrading performance. And it is *much* easier to align in the beginning instead of retrofitting.

  14. Marge’s avatar

    How can you determine if your “virtual machine disk/application partitions have been properly aligned?”

    How can you properly align them to start with?

    How do you keep them aligned?

  15. Jim Nickel’s avatar

    Marge,

    Probably the easiest way to see alignment is to use this:

    http://read.virtualizeplanet.com/?p=366

    vDisk Informer – free product, works great.

    Once a machine is aligned, it will stay aligned (we are talking about partition alignment of the guest OS here).

    Any machine created with Windows 2008 will automatically be aligned correctly.

    Any machine created with Windows 2003 will NOT be correctly aligned. You must correct this – best option is to create an aligned template and then just create more VM`s from that template – then they will all be aligned.

    Jim

  16. paul’s avatar

    write performance of Celerra’s decreases when replicator or snapshots are used. The snapsure technology which is used for both will slow write performance greatly when used. If you get 95-105MB/s write performance initially, you will see this drop to a fraction (as low as 15 or lower MB/s) when writing files to the Celerra. An undocumented but extermely discouraging “feature” of Celerra.

Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>