A niggling doubt about thin provisioned disks was placed in my head when I read Duncan’s article on a Storage VMotion problem; in that article, a statement is made that ESX Server 3.5 no longer supports thin provisioned disks. Intrigued by that comment, I started doing some digging to see if that was actually the case. I was unable to find any concrete statement one way or the other.
Some testing in my lab showed that with ESX Server 3.5, VMDKs are still thin provisioned by default when stored on an NFS datastore. So that put to rest the idea that thin provisioned disks had been abandoned, but now I was curious to follow up on the issue of how ESX Server handled cloning thin provisioned disks, as mentioned in Virtualization Short Take #1.
Additional testing showed that although the VMDKs are indeed thin provisioned at the beginning of their life, they won’t necessarily stay that way:
- Migration of the VMs files from one datastore to another, even if the destination datastore is also NFS, will cause the VMDKs to revert to thick (fully allocated) VMDKs.
- Clones made from the thin provisioned disks have thick provisioned VMDKs.
- A Storage VMotion operation will cause the disks to become fully allocated instead of thin provisioned.
This is a strong counterpoint to the arguments in favor of using NFS for your VMware storage in order to gain thin provisioned disks. In order to really take advantage of thin provisioned disks, every VM must be provisioned from scratch—no cloning within VirtualCenter—and you must give up Storage VMotion or cold migration of the VMDKs.
So far, I have not found any workaround for this behavior. If anyone knows of a workaround, please share it in the comments. (To be honest, I don’t really expect to find one.)
7 comments
Comments feed for this article
Trackback link
http://blog.scottlowe.org/2008/03/31/only-thin-provisioned-in-the-beginning/trackback/
Saturday, August 16, 2008 at 6:05 pm
Pingback from Dispelling Some VMware Over NFS Myths
Monday, March 31, 2008 at 4:45 pm
Matthew
We are currently planning out a data center migration based on this and and we decided to let the NetApp A-SIS de-duplication technology do it’s work here. A thick provisioned disk should de-dupe very well.
The downside is that we need to keep more spare space on the volume in the event a large number of writes, but in our case the high IO volumes are going to be dedicated iSCSI luns or fiber connected to take advantage of SnapManager for SQL.
Monday, March 31, 2008 at 7:37 pm
Andy Leonard
One big problem with A-SIS for VMware that I see is that it is post-process: your LUNs will be fat until the (nightly, or ad-hoc) A-SIS process makes them thin again. Storage VMotion would presumably result in fat vmdks on your destination - and possibly big problems if you oversubscribed your VMFS storage and have to Storage VMotion a lot of VMs in a short amount of time.
Andy
Monday, March 31, 2008 at 10:49 pm
Nick Triantos
Indeed. Cloning or movement of a thin vmdk to another datastore reverts it to thick. This is not just on ESX 3.5 but it happens in the pre 3.5 versions as well. I believe we’ve filed an SR this.
A couple of ways around this when you use NFS:
1) Use Single File Snap restore and clone from a Netapp snapshot and restore a copy with a different name to either the same volume or a different qtree within the same volume
2) Use cp with the sparse option
3) Use ndmpcopy and create a new VM from a NetApp snapshot either onto the same volume on the same array, or a different volume on the same array or across different NetApp arrays regardless of location.
Tuesday, April 1, 2008 at 8:28 am
slowe
Andy,
The fact that deduplication is post-process does present a few challenges in managing it, but I’d rather have it post-process than in-line and causing my performance to take a major nose dive.
Nick,
Thanks for the tips–I’ll have to give those a try.
Tuesday, April 1, 2008 at 9:23 am
Nick Triantos
Given that we enable dedup on primary data rather than just backups and archives, we needed to be cognizant of the fact that performance is still a requirement. Had we done it inline, we’d have the hit “wall” at around 250MB/s TOTAL which the maximum performance achieved today by inline methods today.
So the idea was to achieve a balance between performance and deduplication. Having it behave as a “trash compactor” which can be scheduled rather than a “shredder” allows to obtain the best of both worlds.
Tuesday, April 1, 2008 at 2:07 pm
Andrew Miller
Thanks very much for posting this — I’ve recently found what you’re describing on ESX 3.0.2/VC 2.0.2U1 and was searching to find confirmation.
I’d be very interested in any response to the SR noted above.