Problem with VMXNET3 Driver

A colleague on the EMC vSpecialist team (many of you probably know Chris Horn) sent me this information on an issue he’d encountered. Chris wanted me to share the information here in the event that it would help others avoid the time he’s spent troubleshooting this issue.

What Chris has found is that there is a flaw in Windows Server 2008 and Windows 7 that causes “orphaned NICs” when using the VMXNET3 network driver. There appear to be three cases in which this problem appears:

  1. When you deploy an OVF or OVA of Windows Server 2008 or Windows 7
  2. When you clone a VM running Windows Server 2008 or Windows 7 (this also applies to deploying from a template)
  3. When deploying a vApp within vCD that contains Windows Server 2008 or Windows 7 (this can cause quite a bit of chaos)

Up until now, there were two available workarounds that appeared to resolve this issue:

  1. Use the Intel E1000 driver, which doesn’t cause the same problem. However, it’s unclear how well the E1000 performs with 10 Gigabit Ethernet uplinks.
  2. Use a statically assigned MAC address, which of course doesn’t scale very well in large (and/or dynamic) environments. This is also very difficult to do in vCloud Director (apparently even rising to the level of having to hack the vCD database).

It would appear that the behavior Chris describes is captured in this VMware KB article. There is also a hotfix available in this Microsoft KB article; Chris has tested this hotfix and indicates that it does indeed fix the problem. The referenced Microsoft KB article and the referenced VMware KB article also reference this third Microsoft KB article, further leading me to believe that the articles are indeed related to the same underlying issue.

If you are deploying Windows Server 2008 and/or Windows 7-based VMs in your environment, you might want to take a look at the linked VMware KB and Microsoft KB articles to be sure that you don’t run into the same sorts of problems Chris was experiencing.

Thanks Chris!

Tags: , , , , ,

14 comments

  1. Jason Boche’s avatar

    Thanks for the heads up Chris and Scott. I see Local Area connection #2 in Network Properties quite often. Normally things like this makes the OCD in me twitch but I’ve given up on this as it’s really not all that important in the grand scheme of things unless scripting or some other automated process has hooks into a specific NIC name. Sometimes I’ll rename it manually if I have the extra time.

  2. Chris’s avatar

    The 2 workarounds noted aren’t the only ones available. All that’s needed to correct this is to add a system variable: devmgr_show_nonpresent_devices =1, then go to Device Manager and show hidden devices. You’ll then see the orphaned NICs, and you can uninstall them. You may also need to uninstall any corresponding isatap interfaces as well. This adds a couple extra steps to the post deployment config of the VM, but it’s the cleanest one…

  3. Loren’s avatar

    We ran into this, too, and the patch fixed it for us. It might also be worth noting that SP1 for Windows 2008 R2 requires a patch from a different KB…

    http://support.microsoft.com/kb/2550978

    Cheers,
    -Loren

  4. Mike Karolow’s avatar

    I’ve been wondering about this, thanks for pointing me to the root cause.

    I’ve been solving the problem by uninstalling the NIC prior to shutting down my template VM for the last time. Then, the NIC that generates at first boot is the only NIC. Is there any downside to this approach? (I haven’t found one, yet)

  5. slowe’s avatar

    Jason, I’m not clear if the error that Chris Horn describes is purely cosmetic, or if it actually causes some networking problems.

    Chris, you are correct—there is always the option of cleaning up the orphaned NIC after the fact.

    Loren, thanks for the link to the SP1 version of the patch!

    Mike, I’m not aware of any downsides to your approach, but then again I haven’t tested it extensively, either. Perhaps one of the other readers can provide some feedback. Thanks for commenting!

  6. mkguy’s avatar

    I’ve been using Mike’s approach since vmxnet3 became available with the release of ESX(i)4 and haven’t had any issues with that.

    If it’s just about normal templates/cloning, the “#2″ naming is a purely cosmetic thing and of no importance.

    The real issue which those MS hotifxes are addressing (possible BSODs), could for example happen with streaming Windows with vmxnet3 through Citrix Provisioning Services:
    http://support.citrix.com/article/CTX125361
    http://support.citrix.com/article/CTX129743

  7. ian0x0r’s avatar

    This is also documented on the Veeam website as the same issue occurs if you are running a restore of the VM. It removes any static IP address settings (as they are bound to the orphaned nic). I have had mixed success applying the MS Hotfix, it does work, but it has also removed static IP bindings from exisiting NICS when applying the hotfix to an already running virtual machine, so make a note of your IP address!

  8. Mr. X’s avatar

    Scott,

    Thanks for pointing out this issue. One small correction: Per the VMware article, the issue only affects Windows Server 2008 R2 (and Windows 7) systems, not Windows Server 2008 as stated above. Windows Server 2008 is *not* the same as Windows Server 2008 R2. (You can blame Microsoft for confusing naming conventions.)

  9. daniel’s avatar

    Thanks for the heads-up!

    I think that orphaned NIC’s still retain their static IP addresses, which means that you can’t assign the same IP address to your “new” NIC before the orphaned NIC is removed. I haven’t confirmed this though.

  10. Ken’s avatar

    Thanks for this gents. It helped me solve a problem that I had with ESXi 5.x Dump Collector running on this OS version.

  11. Brian’s avatar

    Old post that Ken already resurrected so i might chime in!

    Anyone seen this happen with E1000 nics? Came accross it a few times lately. The MS patch addresses vmxnet3 and we apply it to all templates but have been seeing a similar issue with OVA/OVF redeploy and ghost E1000 nics on Win2K8 R2 & Win12 lately.

    i would also add that its worth removing the ghost nic interface entry from the registry with the old IP details before reassigning the same IP as i’ve seen this cause major networking issues.

    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces\ID for Adapter

    Cheers

  12. Joëlle’s avatar

    Thank you for this describing post. Among this issue symptoms can the denying of NIC same renaming also bound to the same virtualization behaviour ? If anyone has encountered the error message : “Cannot rename this connection. A connection with the name you specify already exists. Specify a different name.”

  13. Johan de Beer’s avatar

    I downloaded the MS- patch( http://support.microsoft.com/kb/2550978), tried to install it….

    but it says that its not aplicable to my version of windows 2008 r2 SP1.

    Any ideas?

Comments are now closed.