I recently had a colleague contact me with a question about raw device mappings (RDMs) and Storage vMotion. This colleague was trying to perform a Storage vMotion operation on a VM that also had a RDM attached and was running into a problem where the operation was failing because the destination datastore did not have sufficient space. In this case, the free space was less than the size of the mapped raw LUN, and this colleague couldn’t perform a Storage vMotion as a result. The colleague was surprised; he didn’t expect that behavior.
This behavior struck me as odd and unexpected, so I started digging in and doing some testing. I did all my testing on vSphere 4.0 GA code (no updates), thinking that this was probably “worst case”; if anything, the updates would likely resolve any potential issues with RDMs and Storage vMotion. I used a VM with a 20GB VMDK and a 50GB mapped raw LUN, using a virtual mode RDM.
I performed a few Storage vMotion operations and everything seemed fine; I couldn’t recreate the same behavior. Sure, the Datastore Browser shows the VMDK pointer file for the RDM to be the same size as the backing LUN (50GB, in my case), but I couldn’t seem to make vCenter Server balk when migrating the VMDK. I tried several times with datastores of various sizes, including a datastore that had less free space than the size of the mapped raw LUN. Then I noticed something: my 20GB VMDK was thin provisioned. Ah, perhaps that was causing part of the problem. So I performed a Storage vMotion to a larger datastore, selecting “Thick Format” during the process to inflate the VMDK to full size.
Now, had I gone back and carefully re-read the table on page 197 of the vSphere Basic Administration Guide (available in PDF here), I would have remembered that selecting “Thick Format” with a virtual mode RDM automatically converts the RDM to a virtual disk. But I didn’t, and so the RDM was converted into a virtual disk. Subsequent Storage vMotion attempts with the newly-converted virtual disk now produced warnings and errors about available disk space, just as my colleague had seen. Fortunately, this was just a lab environment, so no harm was done. But what if this had been production data?
So here’s the key message I want to convey with this blog post: when you are performing Storage vMotion operations on a VM with at least one RDM and you want/need to convert your virtual disks from thick to thin (or vice versa) during the migration, you need to perform two (2) separate Storage vMotion operations:
- First, you’ll migrate only the virtual disks attached to the VM. You’ll use the Advanced button when selecting the datastore so that you can leave the RDM alone for this migration. You’ll then select the appropriate format (“Thin Provisioned Format” or “Thick Format”) for the virtual disks on the target datastore and proceed with the Storage vMotion operation.
- When the first Storage vMotion concludes, you’ll then perform a second Storage vMotion operation to migrate only the RDMs. Again, you’ll use the Advanced button when selecting the datastore and choose to move only the RDM. For format, select “Same Format as Source”, as this is the only option that preserves the RDM as an RDM. If you are migrating a virtual mode RDM and choose either of the other two options, your RDM is converted to a virtual disk and cannot be converted back.
At this point, your RDM-equipped VM has been migrated to a new datastore, the virtual disks have been converted from thick to thin (or vice versa), and your RDMs have been preserved as RDMs.
Anyone have any other little gotchas like this about RDMs or Storage vMotion they want to share? The comments are open and I’d love to hear any other suggestions or tips from the readers.
Tags: Storage, Virtualization, VMware, vSphere
-
This isn’t particular to vSphere. We’ve run into this on our ESX3.5 farm. Thanks for figuring out the workaround.
adam
-
Perfect, thanks for this Scott, i had the left over RDM pointers sat on a drive and wasnt sure about moving, but following your steps and choosing Same Format as source worked great
-
I ran into the exact same problem just few weeks ago. And i figured out the same workaround. Lost a few hours to find the workaround.
That’s the first time that i said… “hmm what would this advanced button do during storage migration ”I’m sure more people will run into this problem. so thx for blogpost.
Tom
-
Scott, thanks for another great post. With option 2, that takes care of the mapping file and changes DS, But what are your thoughts around process or utilities for also moving the backing lun, such as from one vendor storage to another?
-
JT, there are lots of technologies on the back-end that can help with that. Almost every vendor (including my employer, EMC) has technologies that enable you to move a LUN from one RAID group/aggregate/disk pool to another, and many also have technologies or products to move storage from one array to another. EMC VPLEX, for example, can do that. I believe that NetApp V-Series, IBM SVC, HDS USP-V, etc., can do the same sort of thing, but I don’t know that for certain as I’m not intimately familiar with those products.
Hope this helps!
-
That’s weird – in my experience, if I need to convert RDMs to VMDK disk, I need to shut down the VM, then perform a migrate of the RDM disk and change type to either thin or thick. Trying to storage vMotion a running VM’s RDM disk with thin/thick option results in an error saying: format not supported. And of course, selecting to keep the same format results in RDM pointer files to be moved.
This was on ESX 4.0 update 1.
-
Solgae, the behavior I describe is for a virtual mode RDM. This doesn’t apply to physical mode RDMs, which might be what you are using.
-
Scott, yes within the same storage array there are tools such as NetApp’s lun move command. I’m looking at options to go 3Par to EMC. Adding bonus challenge on top of that is some RDMs are storage-side thin-provisioned wherein it wouldn’t be optimal to block copy into the new arrays; ie I’d like to let the new array do its own T/P. I’ll let ya know what i find.
Thanks
JT -
@slowe: Understood. Indeed they were physical mode RDMs.
-
I’m running ESX3.5 U5, and trying to use the sVmotion plugin to move a VM with a virtual mode RDM attached from one Hitachi storage box (USP100) to another Hitachi Box. (AMS2500) with thin provisioning. I don’t have an advanced option with this plugin, so am I stuck trying to command line move this VM? I haven’t attempted to use the plugin yet, as I noticed this VM had an RDM, and I hit the brakes to research this process before trying it and getting a good backup first.
Thanks -
Try this out for size:
My tested and proven method for migrating and converting physical RDMs
1) log into ESX console as root
2) navigate to your virtual machine folder: /vmfs/volumes/vmfs-volume-freindlyname/vm-folder/
3) locate the pointer file for your disk, this is not the -rdmp file, this is the servername_1.vmdk, it is only a few KBs, inside it does nothing but point to the actual disk file, which in this case will be the servername_1-rdmp.vmdk. It is this servername_1-rdmp.vmdk file that maps IO to the actual lun which is always represented by ESX as something like vml.xxxxxxxxxxxxxxx and can be viewed in /vmfs/devices/disks/
The source you are looking to migrate (with this method it is actually a copy which is good, because the source is always left intact) can be a virtual disk, virtual rdm, or physical rrdm, it doesnt’ matter, you will always use the servername_x.vmdk file as the source in the command string you are about to see…
but before I continune, if you want to migrate from physical compatibility mode lun to a new physical compatibility mode lun, be sure to have provisioned the lun so ESX can see it. If you are to convert the physical mode RDM to virtual RDM, the same applies. If you are to convert it to virtual disk, just be sure you have a vmfs volume with enough space to accomodate the new disk.
4) here it is:
To migrate and convert physical RDM to virtual disk:
vmkfstools -i servername_1.vmdk /vmfs/volumes/vm folder/servername-newdisk_1.vmdk
To migrate and convert physica RDM to virtual RDM, there are two methods
- in vi client edit VM settings of powered down VM, remove rdmp mapped disk, then re-add a new hard disk selecting the same disk only be sure to select virtual as the mode, you will log in and see all of your data and mount points are unchanged.
- from the ESX prompt in the virtual machine folder on the VMFS volume:
vmkfstools -i servername_1.vmdk -d rdm:/vmfs/devices/disks/vml.xxxxxxxxxxxxxxxxxxxxxxxx /vmfs/volumes/vm folder/servername_1-rdm.vmdk
To migrate physica RDM to a new physical RDM:
vmkfstools -i servername_1.vmdk -d rdmp:/vmfs/devices/disks/vml.xxxxxxxxxxxxxxxxxxxx /vmfs/volumes/vm folder/servername_1-rdmp.vmdk
Command strings EXPLAINED:
vmkfstools -i is the import \ clone command
the servername_x.vmdk is the source, the x will stand for the order of the device, in my example I use 1 which means this is the second disk on the VM. the first disk file is servername.vmdk, this is typically always the system disk.
the -d is the disk type for the destination you are converting to
the rdm(p) is part of the disk type telling the destination disk what the raw disk mode is, virtual or physical (the p means physical obviously)
the /vmfs/volumes/vm folder/servername_1-rdm.vmdk tells us the location of where to store the raw disks mapping file, I have chosen to put it in the virtual machine folder as you can see, you can put it on any vmfs file system, I like to keep things neat and clean.
Figuring out which vml.xxxxxxxxxxxxxxx is tricky.. I the ls-lh command on the /vmfs/devices/disks/ directory to list the unfriendly volume names out and figure out which one is mine by the LUN size, there are other ways to do this, there is a VMware KB article that describes other ways to dump out this information. and in some cases, depending, there may be a partition involved and it would look like vml.xxxxxxxxxxxxxxxx:1 or :2 or whatever.. vmware’s KB references it as vml.xxxxxxxxxxxx:p
you can also use vmhbax:x:x: in the string as such: /vmfs/devices/disks/vmhbax:x:x:p to represent the raw LUN
check out this article, it has what you need, but not everything.. the example they use for the disk is naa.xxxxxxxxxxxxxxxxx and I believe this is because there is different storage in use then I have… SO, be sure to navigate to /vmfs/devices/disks/ to see how vmware lists out the disks as the storage presents them
Hope you find this useful, don’t let anyone tell you that you can’t migrate a physical RDM. Its bogus information. This is how its done.
-
If I am not confused, which is a regular thing in my case, in your last paragraph, this process only migrates the mapping file. You cannot actually migrate an RDM to a new datastore without downtime, unless you do as you say and convert to thin/think vmdk. Migrating the mapping file leaves the RDM on the existing storage and the mapping file now with the VM on the new datastore. You will still have to sync with the native storage tools and then detach and attach the RDM.
-
SamuraiJack, I’m not sure if you’re commenting to me or one of the other readers, but I’ll have to take issue with the wording here. “Migrating an RDM” *can* be done live, as “migrating an RDM” is the same as “migrating the pointer file”. An “RDM” is a raw device mapping, i.e., a pointer file to a raw LUN. You can’t migrate the backing LUN in vSphere (you’ll need to resort to array-level tools for that). A silly matter of terminology, I know. Still, your point about migrating the backing LUN is a point well-taken! Thanks for your comment.
-
This issues is still there in vSphere 5, and they took away ability to set Scsi.CRTimeoutDuringBoot = 1.
Only workaround I found was:
esxcli storage core device setconfig -d naa. –perennially-reserved yesbut that required me to find the LUN ID’s for each RDM, and have the host fully booted once.
Credit for the command: http://communities.vmware.com/thread/327679
-
I have a Suse Linux cluster with three nodes and physical RDMs that I want to make an Storage vMotion on. Is it still the same solution for that?
-
Hi,
I try the steps above. It will work with the ‘normal’ virtual disks’, but when I try to migratie the RDM pointer file I receive a error message after 95% of the migration like “incompatible device backing specified for device ’0′” remove the CDROM and Floppy didn’t resolve the problem:-( any suggetions
Site Sponsors
16 comments
Comments feed for this article
Trackback link: http://blog.scottlowe.org/2010/08/18/storage-vmotion-with-rdms/trackback/