Another Reason Not to Use PVSCSI or VMXNET3

You might have read the article I wrote here titled vSphere Virtual Machine Upgrade Process, in which I described a process whereby you could upgrade your VMs to VM hardware version 7 (the version used with vSphere) as well as use the latest paravirtualized network and SCSI drivers (VMXNET3 and PVSCSI). Both PVSCSI and VMXNET3 offer greater performance with the same CPU utilization.

Rightfully so, some readers and other bloggers pointed out that PVSCSI isn’t supported for boot disks (Rich Brambley put up a really good post, for example). Rich, among others, suggested moving virtual machines back to a “two disk model,” with a boot disk and a separate data disk; this would allow for the greater performance of the PVSCSI controller on the data disk. This seemed to be a reasonable workaround. I don’t recall hearing about any significant issues with VMXNET3. Using the newer network driver seemed to be a good move all the way around.

Unfortunately, there is another drawback to both of these devices. Rich caught this drawback in his article, but relegated it to a small mention at the very end of the article that even I overlooked at first (emphasis mine):

There are some other factors to consider as well. For example, vSphere Fault Tolerance cannot be enabled on a VM using PVSCSI.

That’s right—you cannot use VMware Fault Tolerance (FT) on a virtual machine that is using the PVSCSI device. However, this restriction doesn’t just apply to the PVSCSI device; it also applies to VMXNET3! VMware FT cannot be enabled on a virtual machine using either the VMXNET3 or PVSCSI devices; vCenter Server will simply report an error that the network interface or disk controller isn’t supported for VMware FT.

In my opinion, this is a significant enough limitation that I felt it warrants its own post. If you are planning on using VMware FT in your environment, be sure not to configure any virtual machines to use VMXNET3 or PVSCSI if they might need to be protected with VMware FT. In this case, you’ll have to choose from either maximum performance or maximum protection—you don’t get both.

UPDATE: Rich Brambley shared links to two resources that describe the incompatibility between VMware FT and PVSCSI and VMXNET3:

VMware Communities: Unable to configure FT with error “Unsupported virtual machine configuration for Fault Tolerance. Device ‘Network adapter 1′ is not supported”
VMware Fault Tolerance Requirements and Limitations

Tags: , , ,

  1. william bishop’s avatar

    crap!

  2. william bishop’s avatar

    Sorry, that was my visceral instant response to this, since I’ve already been building new templates for our migration process.

    Now to go undo it.

  3. rbrambley’s avatar

    Scott,

    Some VMTN Communities links for your readers on these FT limitations-

    Error received when VMXNET3 in use:
    http://communities.vmware.com/thread/217845

    FT requirements and limitations:
    http://communities.vmware.com/blogs/vmroyale/2009/05/18/vmware-fault-tolerance-requirements-and-limitations

    Thanks for the reference to vmetc.com!

  4. James’s avatar

    Thank you for the information Scott. Keep up the excellent work.

  5. stu’s avatar

    I think this post would be better titled “Another reason not to use FT”, given the already extremely limited use cases for FT.

  6. Brad Hedlund’s avatar

    Scott,
    I’m wondering if the FT and VMXNET3 issue is a bug or otherwise something that can be quickly fixed in a minor release? Any insights into that?

    Thanks,
    Brad

  7. Igor.Nemilostivy’s avatar

    Hello!
    “be sure not to configure any virtual machines”
    Sorry, why for all. Only for those what planing put to the FT. Right?

  8. Scott Vessey’s avatar

    It makes sense that VMXNET3 (which can be used with VMDirectPathIO) and PVSCSI don’t allow for FT – any VM using those devices is “tied” to that ESX host.

    It’s the same principle as not allowing VMotion of a VM with a CPU affinity set, or of a VM with it’s CD-ROM connected to an ISO image on the local VMFS.

  9. slowe’s avatar

    Rich,

    Thanks for the links (I’ve added them to the article) and for your great work!

    Stu,

    You are correct-in its current incarnation, VMware FT is incompatible with just about every other feature in VMware vSphere. I’m quite confident that VMware will address that in upcoming versions.

    Brad,

    I believe that the VMXNET3 issue is a bug; the VMware Communities thread to which I linked in the article has a response by a VMware employee that implies it is a known bug that will be addressed.

    Igor,

    Yes, of course–you could most certainly leverage PVSCSI and VMXNET3 on any virtual machines that will not be protected via VMware FT.

    Thanks to all who commented!

  10. slowe’s avatar

    Scott Vessey,

    Although these drivers allow hypervisor bypass via VMDirectPath, that should not preclude their use with VMware FT. At least, that’s my humble opinion.

  11. Scott Vessey’s avatar

    Scott L,

    I think we’re all expecting many of the current limitations of FT to be relaxed in a future release, I would imagine this is when we’ll get separation between FT and VMXNET3. It could be an oversight in the current release, it could be that it wasn’t possible prior to GA date….

    Scott V.

  12. Ronghua’s avatar

    Scott V,

    VMXNET3 currently does not support VMDirectPath yet, so VMs using VMXNET3 are not tied to the physical host.

    Virtual devices need enhancement to support FT and such enhancement does not exist in VMXNET3 yet, that’s why you cannot enable FT when the VM is using VMXNET3.

  13. Michael McNamara’s avatar

    Scott,

    Thanks for taking the time to share your knowledge. As a person responsible for the successful deployment of a virtual initiative I’m grateful to have resources like yours where I can get up to speed fast on the ups and downs of this technology.

    While there’s no substitute for trial and error on the actual equipment, it’s very helpful to have a solid idea of where we’re going and how things will work.

    Cheers!

  14. Iben Rodriguez’s avatar

    Stu, I’m with you.

    This post should be renamed…

    More limitations of FT.

    All our VM guests hosted on VI3 are being switched over to VMXNET2 for improved performance.

    As we move guests to new vSphere clusters the vmware tools are being updated as well as the NIC types being changed. We’re considering updating the SCSI driver type too. Some guests still have buslogic and are being change to LSI Logic. I noticed now there are multiple choices, SAS, SCSI, etc…

    For the few use cases where FT is needed or desired seems like it would be fine leaving the NIC at VMXNET Enhanced (VMXNET2).

    Any thoughts on the performance advantages (or disatvantages) of the following SCSI controller types:
    BusLogic Parallel
    LSI Logic SAS
    LSI Logic Parallel
    VMware Paravirtual

  15. Todd Williams’s avatar

    Scott – Have there been any updates from VMware since you originally posted this article to fix the incompatibility between VMware FT and PVSCSI and VMXNET3?

    Thanks again for the good info and I’m glad that I read this when moving to vSphere as I’ve decided to stick with VMXNET2 (Enhanced) for the time being.

  16. Michael’s avatar

    Although maybe not “supported” something else to note is the VMXNET3 Adapter does not allow the MAC address to be specified via the Windows Guest OS Driver.

  17. Iben Rodriguez’s avatar

    Hi Scott,

    With the latest release of ESX 4.0 Update 1 you can now use PVSCSI for the boot drive.

    Who uses FT anyways?

    We have customers actively upgrading their machines to vmxnet3 and PVSCSI now.

    I b e n

  18. Jack’s avatar

    Just to update this post, VMXNET3 supports FT now in ESX 4.1.

  19. Jack’s avatar

    also PVSCSI supports FT as well in ESX4.1, so both impossibilities solved.

    27.ParaVirtual SCSI
    VM configured with a PVSCSI adapter can be part of an Fault Tolerant cluster.
    PVSCSI adapters already support hot-plugging or hot-unplugging of virtual devices, but the guest OS is not notified of any changes on the SCSI bus.
    Consequently, any addition/removal of devices need to be followed by a manual rescan of the bus from within the guest.

  20. Tramadol’s avatar

    Have there been any updates from VMware since you originally posted this article to fix the incompatibility between VMware FT