vSphere Virtual Machine Upgrade Process

Upgrading a VMware Infrastructure 3.x environment to VMware vSphere 4 involves more than just upgrading vCenter Server and upgrading your ESX/ESXi hosts (as if that wasn’t enough). You should also plan on upgrading your virtual machines. VMware vSphere introduces a new hardware version (version 7), and vSphere also introduces a new paravirtualized network driver (VMXNET3) as well as a new paravirtualized SCSI driver (PVSCSI). To take advantage of these new drivers as well as other new features, you’ll need to upgrade your virtual machines. This process I describe below works really well.

I’d like to thank Erik Bussink, whose posts on Twitter got me started down this path.

Please note that this process will require some downtime. I personally tested this process with both Windows Server 2003 R2 as well as Windows Server 2008; it worked flawlessly with both versions of Windows. (I’ll post a separate article on doing something similar with other operating systems, if it’s even possible.)

  1. Record the current IP configuration of the guest operating system. You’ll end up needing to recreate it.
  2. Upgrade VMware Tools in the guest operating system. You can do this by right-clicking on the virtual machine and selecting Guest > Install/Upgrade VMware Tools. When prompted, choose to perform an automatic tools upgrade. When the VMware Tools upgrade is complete, the virtual machine will reboot.
  3. After the guest operating system reboots and is back up again, shutdown the guest operating system. You can do this by right-clicking on the virtual machine and selecting Power > Shutdown Guest.
  4. Upgrade the virtual machine hardware by right-clicking the virtual machine and selecting Upgrade Virtual Hardware.
  5. In the virtual machine properties, add a new network adapter of the type VMXNET3 and attach it to the same port group/dvPort group as the first network adapter.
  6. Remove the first/original network adapter.
  7. Add a new virtual hard disk to the virtual machine. Be sure to attach it to SCSI node 1:x; this will add a second SCSI adapter to the virtual machine. The size of the virtual hard disk is irrelevant.
  8. Change the type of the newly-added second SCSI adapter to VMware Paravirtual.
  9. Click OK to commit the changes you’ve made to the virtual machine.
  10. Power on the virtual machine. When the guest operating system is fully booted, log in and recreate the network configuration you recorded for the guest back in step 1. Windows may report an error that the network configuration is already used by a different adapter, but proceed anyway. Once you’ve finished, shut down the guest operating system again.
  11. Edit the virtual machine to remove the second hard disk you just added.
  12. While still in the virtual machine properties, change the type of the original SCSI controller to VMware Paravirtual (NOTE: See update below.)
  13. Power on the virtual machine. When the guest operating system is fully booted up, log in.
  14. Create a new system environment variable named DEVMGR_SHOW_NONPRESENT_DEVICES and set the value to 1.
  15. Launch Device Manager and from the View menu select Show Hidden Devices.
  16. Remove the drivers for the old network adapter and old SCSI adapter. Close Device Manager and you’re done!

If you perform these steps on a template, then you can be assured that all future virtual machines cloned from this template also have the latest paravirtualized drivers installed for maximum performance.

Post any questions or clarifications in the comments. Thanks!

UPDATE: Per this VMware KB article, VMware doesn’t support using the PVSCSI adapter for boot devices. That is not to say that it doesn’t work (it does work), but that it is not supported. Thanks to Eddy for pointing that out in the comments!

Tags: , , , ,

  1. Eddy’s avatar

    Nice, except that VMWare doesn’t support booting from a PVSCSI device, so you might want to think twice before performing step 12 on your boot disk . . .

    from http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1010398

    “Note: Booting from a disk attached to a PVSCSI adapter is not supported. The system software must be installed on a disk attached to an adapter that does support bootable disk.”

  2. slowe’s avatar

    Ah, I hadn’t seen that KB article yet. Good find, and thanks for pointing it out!

  3. Tom’s avatar

    Regarding: UPDATE: Per this VMware KB article, VMware doesn’t support using the PVSCSI adapter for boot devices.

    Does this mean that one should only use PVSCSI for non-boot disks that only contain data, logs, etc.?? to be sure of receiving VMware support etc.??

    Thank you, Tom

  4. slowe’s avatar

    That is how I read it, yes.

  5. Duncan’s avatar

    That’s how it is Tom!

  6. Sean’s avatar

    This looks like a lot of work and a lot of scheduled downtimes for a site with hundreds of VMs. Do you know what problems might arise for sites that leave some or most of their VMs (that were created on 3.5) not upgraded and only have newer VMs using the new virtual hardware?

  7. kyle’s avatar

    that really is interesting – dont use this for your boot drive. but we will support you using this for a data drive. What the heck is that? Why even offer it now if its not production ready. Vmware has me really shaking my head on this one.

  8. Austin Workman’s avatar

    For those upgrading VMware tools in a Linux environment(Especially Oracle NFS shops!!!). Make sure to backup your /etc/fstab file before applying the upgrade. Went to upgrade vmware tools while I was upgrading the hardware versions to 7 in our dev/test environments and wiped out all of the nfs mount paths for our Oracle DB servers. Each VM had like 8 mount path’s too! It’s a good thing we started with vSphere in dev/test and didn’t jump the gun.

    http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1006718&sliceId=1&docTypeID=DT_KB_1_1&dialogID=24732296&stateId=1%200%2024734213

  9. Dave O.’s avatar

    Scott, above you say have to save the IP info from the old NIC and replace the IP info on the “new” VMXNET3 adapter.

    Here’s an easy way to dump the IP info to a text file and then reapply it

    To dump the IP config using netsh from a command line:
    netsh interface ip dump > c:\ipconfig.txt

    Since Windows will most likely see the new NIC as “Local Area Connection 2” (or something similar) you have to modify the above text file and change the NIC name to match the new NIC’s name. Or change the new NIC’s name on the host to match what’s in the file above. Either way works.

    To re-import it:
    netsh -c interface –f c:\ipconfig.txt

    This really comes in handy when you have a lot of DNS servers, WINS servers, etc and/or multiple IPs on the same NIC.

  10. Rupen S.’s avatar

    ALWAYS make sure that if you are doing this manually (and not using VUM), that you take a snapshot of the VM before the hardware upgrade (from v4 to v7) – else you will not be able to revert back if things go wrong.

    So for the manual steps above, I would insert step 3.5 – Take a snapshot of your VM

    VM upgrade (VMware Tools and VM hardware) are the only upgrade steps that WILL require downtime, and if you have a large number of VMs – it is best to use VUM.

    There are certain vSphere features that will require v7 hardware, like
    - FT
    - HA Improvements (VM Monitoring)
    - hot add/remove virtual devices
    in addition to vmxnet3, pvscsi

    And remember to make sure that you reboot your Windows VMs after the hardware upgrade if it has not been rebooted.

  11. Scott McD’s avatar

    Are these steps included in any new Best Practice or kb article? If we were to just follow the Vmware ‘s upgrade video

    http://download3.vmware.com/vsphere/vsphere-migration-part3.html

    there is no mention of adding any new devices like Scott describes.

    Just so that I am clear on the whole process. You are adding the new devices after upgrading the Virtual Hardware so that the guest OS will load the new drivers?

    Scott from your testing did you find you could select ParaVirtual SCSI before without adding and removing a dumby drive or does this somehow effect the “unsupport” PVscsi boot drive?

  12. slowe’s avatar

    Scott McD,

    The purpose in adding the dummy drive is to get Windows to load the PVSCSI driver. Then, when you switch the boot controller, the driver is already in the guest OS’ list of drivers to load. Otherwise, you’ll get a BSoD (INACCESSIBLE_BOOT_DEVICE).

    And yes, you will add the new devices AFTER upgrading the VMware Tools and AFTER upgrading the virtual hardware.

    Hope this helps!

  13. Patrick’s avatar

    Hmm, luckily I plan on redoing my complete setup, as usual great stuff Scott.

  14. Dan’s avatar

    @Dave O
    There is a syntax error in you netsh command to restore.
    Should read as this: “netsh –f c:\ipconfig.txt”

    Here is the KB on doing this when threre are lost configs on VM’s

    http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1005595

  15. sandy84’s avatar

    Realy helpfull

  16. Matt’s avatar

    Hey everyone,
    Maybe I’m missing the boat here, but on step 12, when I create a Hard Disk, it doesn’t ask me anywhere in there to select the adapter type. Instead I just have a single checkbox on the options tab to generically enable or disable paravirtualization for the whole VM.

    My VM has the updated tools, Hardware v7, and ESX version is 4.0.0 (175625), which should be latest and greatest. What am I missing? Thank you!

  17. Matt’s avatar

    Bah… If only I could read. I was making it 0:1, not 1:0 like it was pointed out in another article. Now that I brought up VMI paravirtualization (above), does anybody have any thoughts on it? VMware says it’s supposed to work great. Any drawbacks? Thanks.

  18. dave’s avatar

    Thanks a lot! it’s been a great help for my esxi 3.5 to vsphere migration

  19. David Markley’s avatar

    Scott, thanks a lot for this concise step-by-step article. I have used this many times in the last several weeks as customers are migrating to vSphere. I have been using PVSCSI with no problems for quite a while, even on boot disks. I can understand why VMware would not want to support this, however we used NFS before ‘official’ support and Guest VM monitoring in HA before ‘official’ support. So, I guess we just like to live on the wild side. *grin*

    Also wanted to say that I built just a simple batch script that gives a menu with common options like Import/Export network settings, start Device Manager with non-present set to 1, immediate Shutdown/Restart options, etc.

    It saves a bit of time when you’re doing many of these types of migrations.

    http://www.davidmarkley.com/vmware/guest-vm-settings-migrator-1-0

    Should save a few minutes when doing this stuff…

  20. Fabio Rapposelli’s avatar

    Don’t know if it has been updated, but right now the KB article doesn’t state anywhere that PVSCSI is not supported on boot disks.

  21. Chris Waltham’s avatar

    Does anyone know how to upgrade the virtual hardware without the VM’s MAC addresses changing?

  22. Scott Sauer’s avatar

    I agree with Sean, what’s the downside to doing nothing, and only using the new 7 format going forward? Just newer feature support? We have a large environment and it would take months to do all system.

  23. Jim Nitterauer’s avatar

    The restore syntax for the netsh command listed above should be:

    netsh –f “c:\ipconfig.txt”

  24. Marc’s avatar

    On the topic of the PVSCSI driver, it should also be noted that PVSCSI is only supported on Windows 2003, Windows 2008 and Redhat Enterprise Linux 5. Furthermore, it is not supported for boot disks, record/replay functionality, fault tolerance and MSCS clustering. VMware recommends to use PVSCSI only on high performance shared storage.

    If you use PVSCSI, there are some futher limitations you should be aware of:
    - doing a “hot-add” or “hot-remove” requires a bus rescan;
    - there may not be any performance gain when there are snapshots attached to the disk;
    - there may not be any performance gain when the host memory is overcommitted;
    - on a RHEL5 system, after a kernel upgrade you will not be able to “see” you data until after you run “vmware-config-tools.pl”.

    For full details, see page 166 of the vSphere Basic System Administration Guide.

  25. Craig Taylor’s avatar

    Hi guys,

    as we run a reasonably large VM farm, we are looking for ways to automate the safer parts of this upgrade. As such, I have written a script which upgrades the vNIC of all legacy guests to VMXNET3, with error handling for the relevant errors (e.g. incorrect tool version etc). Would a link in this comment thread be appropriate? Thought I’d ask first…

  26. kyle black’s avatar

    Hi guys,

    In all my Vmware readings I’ve never found any information on re-doing your HDD and NIC drivers on all your Vm’s. Is this really required? Or a “just a nice to have feature enabled?”.

    We don’t do any paravirtualization. I was under the assumption the hardware upgrade did the actual upgrade, again based on my readings.

    Can anyone confirm that unless I use paravirtualization that adding and removing hdd’s and nic’s from windows 2000/2003 VM’s to enable the v7 drivers isn’t required?

  27. slowe’s avatar

    Kyle,

    Make no mistake, you are using paravirtualization anytime you install the VMware Tools into one of your guest operating systems. VMware Tools uses paravirtualized device drivers—meaning device drivers that are virtualization-aware—in order to optimize performance. The guest OS kernel might not be virtualization-aware, but the VGA, network, and (maybe) disk device drivers are.

    You only have to follow the NIC/HDD drivers portion if you want to use the latest VMXNET3 and PVSCSI device drivers, which are intended to provide the highest levels of performance with the lowest amount of overhead. If you don’t want/don’t need to switch to VMXNET3 and/or PVSCSI, then just upgrading VMware Tools and then upgrading the VM hardware is enough.

    Hope this helps!

  28. kyle black’s avatar

    Thats helps! Thanks so much for your reply! When I create new Virtual Machines / Templates I assume the latest drivers will then be employeed by default I hope! I wonder after a person p2v’s new machines if the new drivers (*pvscsi and vmxnet3 * ) are installed then when you first boot the machine and install vmtools!

    Thanks again!

  29. TomP’s avatar

    Craig,
    I for one, would be interested in a link to your script.
    Thanks

  30. Art Vinogradsky’s avatar

    Please note that PVSCSI on boot disks is now officially supported with u1.

    From Vmware U1 release notes:
    Enhanced VMware Paravirtualized SCSI Support – Support for boot disk devices attached to a Paravirtualized SCSI ( PVSCSI) adapter has been added for Windows 2003 and 2008 guest operating systems.

  31. PiroNet’s avatar

    Hi Scott, a situation where there is no vSphere client and no Web Access, is there a command line to do the virtual hardware upgrade?

  32. Brian A’s avatar

    Do note that for now PVSCSI is only intended for high-IO environments.

    Latency may suffer if you are not pushing the IO subsystem fast enough.

    http://vpivot.com/2010/02/04/pvscsi-and-low-io-workloads/

  33. Jonathan Reininger’s avatar

    Scott.

    I just added two disks to a Windows 2k3 VM which C:\ (boot) was on a LSI controller. SO I added one disk on an LSI scsi controller and one disk on the paravirtual scsi controller. I did some benchmarking (ok a .vbs script to copy a 50g folder to the same drive five times).

    SO then I tough, oh Ill just use the iso boot converter to V2V my VM and change its C:\ disk to a paravirtual scsi controller. Only when I booted up the VM I saw just 2 disks (its C: and the new LSI scsi disk).

    Is the disk which was ‘hung’ off the paravirtual scsi controller did seem to be there. Is this NOT supported by the converter boot cd? I assume converter boot CD has no ‘drivers’ for this new scsi controller. In some cases, I use the converter to v2v VMs to change the virtual disk sizes, etc (granted w/ Thin Provision it will change this up a bit).

    This might be a deal breaker for some?… Can anybody thik of any other drawbacks w/ this limitation? Anybody else want to confirm this?

    Jonathan Reininger

  34. Jonathan Reininger’s avatar

    ^^ btw, at times I (and others?) v2v a VM to adjust the boot drive c:\ size (only) but if it was a paravirutal disk I might not be able to use this method. correct?

    (please append to my pervious post)..

    http://www.four2.org/reininger.pdf

  35. Alexandre Pitre’s avatar

    Craig, any chances we could get your script ?

  36. Olu’s avatar

    Hi Scot,

    In an environment with over 3000 vm (combination of windows and linux) just upgraded to vsphere, do we have to all these? (345 Hosts)
    Also can we install vm tools and schedule reboot with windows update?
    My great conern is the hardware upgrade?

    Can you please advise?

    Many thanks

  37. slowe’s avatar

    Olu,

    You can deploy VMware Tools in other ways (via existing software distribution methods), but I don’t think that Windows Update will work.

    As for the VM hardware upgrade, treat it just like you would treat any other project—plan it out, schedule it, and then run it. You can use vCenter Update Manager baselines assigned to various objects within the hierarchy to help with that process.

    Good luck!

  38. Donald’s avatar

    it appears that you can now put Windows boot disk on a PVSCSi. You must be on ESX 4.1 and Linux is still not supported. See the KB in the link

    http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1010398

  39. Brandon’s avatar

    Just a reminder that since Update 1, using pvscsi for booting on Windows supported OSes is now OK. Your method of using a second scsi disk also adds an additional step for those who are cautious but there are ways to “inject” the pvscsi driver (scripts, google is your friend) before shutting down after the initial vHardware upgrade. That way you can simply flip the first adapter over without having to create / delete the other.

  40. Henk’s avatar

    Did it on VMWare 4.1, works fine so far, even on boot volume of W2K3ENT

  41. edgar’s avatar

    hi,

    i have a question

    it is normal for the upgrade hw 7, the network card is configured as a removable device like a usb, in the S.O. Windows server 2003.?

    tnks.

  42. Jose’s avatar

    Hi all,

    I couldn’t get the AMD PCnet adapter (vmnet.sys) removed from our W2K3 R2 Std. Server even while I did the SET DEVMGR…. thig before on the command prompt. The device manager does not show up this adpater any more so I did a search-n-del trough the registry and removed all vmxnet.sys from all ControlSets and also the driver file intself in …\system32\drivers\.

    The system seams to run ok with this and I also set a Snapshot before doing all this. I did not change the SCSI adapter type as it is only a domain controller with not much disk activity.

    Anyhow I wonder why we delete the AMD PCnet adapter before adding the new one. Why don’t have two installed for a short while?

    I also found this KB article related to ESX 3.5 (I know this is out of focus):
    http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1007195

    Why does VMware change the Windows 2K3 R2 Std to Enterprise for doing this? Does VMware Tools change behave on an Enterprise OS??

    If you have a comment please let me know. I’ll give it a try now and see what VMwaretools does when I change the OS to W2K3 EE.

    BTW: The initial reason while I came along was that VMware ESX 4.1 complains while changing the host of a running VM with DRS, HA or vMotion while HW version was still 4. The vSphere event log will show “insufficient video ram” and no chance is give to set the video memory to”auto-detect” as recommended from another KB article for fixing this.

    Now that I have HW vers. 7 and set video ram to “auto-detect” vMotion is running fine without any complains.

  43. Balu’s avatar

    Hi, I haven’t come across this page before i started the
    work of upgrading the Live vm’s.I’ve followed vsphere upgrade guide
    and continuing with the project now but we have met different
    issues.One similar issue as Austin has pointed out is with
    upgrading servers with SQL which has vm tools 3.5 installed. refer
    this kb and make sure to backup the mentioned dll file.
    http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1012259
    Also snapshot should be taken prior to any change.

  44. slowe’s avatar

    Balu, that’s great information—thank you for sharing it with everyone!

  45. Balu’s avatar

    Another update: i did vm upgrade on ISA 2006 and it was tricky.The static routes configured on RRAS disappeared during upgrade.In one of the interface, i was able to give old mac but didnt work on other. But that doesn’t seem to cause any issues now.Added the static routes and working fine now. Also, for configuring vmxnet3 adaptors, i need to delete 4 old from Hidden adaptor list.( 2 old+2 new created during vm hardware change).
    So normally, when you want to add one vmxnet3 adaptor, you need to delete 2 old adaptors.

  46. Osman Nuri’s avatar

    Hi Scott,
    Really great guide. Thanks for sharing this. The only thing is not clear on my mind is “Remove the drivers for the old network adapter and old SCSI adapter. ” When i try to remove hidden files it says “Failed to uninstall the device. The device may be required to boot up the computer.” for all old devices. How to understand which devices should be uninstalled?

  47. slowe’s avatar

    Osman, the drivers that you need to remove should be hidden drivers, which is why you need to show the hidden devices feature within Device Manager. It should be fairly clear which devices are active and which ones can be removed.

    Good luck!

  48. JohnR’s avatar

    Craig,

    Any chance you still have that script handy?

  49. Ravi Shanghavi’s avatar

    A script would be very welcome for this task. Thanks!

Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>