Part 1 in our series on NetApp FlexClones and VMware discussed in greater detail some of the advantages of using FlexClones for VM provisioning. In that article, we saw that using FlexClones can greatly reduce both the storage required for new VMs as well as the time required to provision new VMs, especially when the storage needed by the VMs is large. Both of these advantages can be very compelling.
However, in order to make an informed decision about whether we should use FlexClones we must also look at the disadvantages of this approach. In this part of the series, we’ll take a look at some of those disadvantages.
Just as there were two key advantages to using FlexClones, there are also two key disadvantages to using FlexClones:
- First, the use of FlexClones (unless properly architected) may cause installations to bump up against the maximum number of LUN IDs available (255), or the maximum number of LUNs that may be opened concurrently by all virtual machines (256). (Both of those numbers were taken from the “Configuration Maximums for VMware Infrastructure 3†white paper, available here.) Unless properly architected, this means that your installation could be limited to about 250 virtual machines. Note that there are ways around this limitation; see below.
- Second, there is currently no integration from either Network Appliance or VMware to help automate the process of using FlexClones for VM provisioning. As shown in my technical article on how to use FlexClones for VM provisioning, there are a number of manual steps currently, and users seeking a greater level of automation must create their own scripts or hope that someone else already has written scripts they can re-use.
While these are the biggest disadvantages (in my opinion) there are also a number of other, smaller disadvantages as well:
- Using FlexClones for VM provisioning blurs the operational responsibilities of the SAN administrators and the server administrators; SAN administrators are now responsible for provisioning new VMs via FlexClones.
- Using FlexClones for VM provisioning adds complexity to the solution, making it harder to troubleshoot and administer over time.
- Making the most of the FlexClone space-conserving functionality requires more specialized VM configuration, such as dedicated VMDKs for pagefiles/swap partitions on separate LUNs.
These limitations can really derail the use of FlexClones in larger deployments. Ironically, it’s the larger deployments where the advantages of FlexClones are the most significant. In larger environments, there are typically dedicated SAN administrators and dedicated VMware/VI3 administrators. Introducing the use of FlexClones in this type of environment now begs the question: who’s responsible for provisioning VMs?
While some of these disadvantages are inherent in the solution, there is a workaround for one of them, and that’s the maximum number of LUN IDs/LUNs. Because each FlexClone is a separate LUN with a separate LUN ID, your installation will be limited to about 250 LUNs. If you only place one VM per LUN, then you’ll be limited to about 250 VMs. However, if we instead take the practice of placing multiple VMs—each of them prepared for cloning as described in my how-to article—then we can easily scale the number of VMs. For example, we could create a master VMFS datastore that had 10 VMs built and prepared for cloning, then deploy FlexClones for each group of 10 VMs that were needed. Using this technique, it would be reasonably easy to scale this solution to support 2,500 VMs (250 VMFS datastores with 10 VMs each). (Of course, 1,500 VMs is the configuration maximum for a single VirtualCenter management server.)
In summary, the use of FlexClones for VMware environments has some very compelling advantages, and some very significant disadvantages. It will be up to each organization to weigh these pros and cons against the specifics of their company to determine which route to take. At least now you have some information upon which to base your decision.
As always, I welcome any comments below.
Tags: NetApp, Storage, Virtualization, VMware
-
Scott,
Surely a flexclone is just a read witeable snap of the original source LUN and not a clone at all. That would mean that all of the flexclones although initially space efficient, would effectively be competing for the same disk resource and in many cases the same data blocks. As they divergeover time then the competition for the same blocks may decrease whilst the capacity requirements would increase, although they would still effectively be competing for the same disk resource. The other issue is the fact that this is just a snap which means it’s completely reliant on the parent LUN’s integrity for it’s survival in a failure. Assuming the original source LUN fails for some reason then you also lose all of the flexclones.Cheers
John -
John,
A FlexClone is a writable snap of the original volume (not necessarily the LUN, which is contained in the volume). Over time, as the number of changed blocks in the FlexClone increase, then the FlexClone becomes increasingly less space-efficient as additional block must be allocated to store the changes. This is true for *any* storage system that allows snapshots; the difference here is that NetApp is providing writable snapshots.
As for contending for disk resources, the FlexClone will contend with other workloads on the storage system for the disks in the aggregate; but again, that is no different than any other storage system. Any time you serve multiple LUNs or volumes off the same physical disks, you have to be aware of physical disk contention.
Third, as for contending for data blocks, this is not how WAFL works. Unlike some other storage systems where storing changed blocks after a snapshot involves a “copy-on-write” process to move the original data, Data ONTAP (the operating system used by NetApp) and WAFL (Write Anywhere File Layout; the file system used by Data ONTAP) will simply place the changed blocks of the FlexClone onto the next available area on the disk and update any necessary pointers. This is what allows NetApp storage systems to store up to 255 Snapshots per volume without noticeable performance degradation. This is also what allows the FlexClones to have snapshots themselves, as well.
Finally, with regards to losing the original LUN: you would have to lose more than 2 physical disks in a RAID group (since NetApp, by default, uses RAID-DP with two parity drives) before an aggregate, volume, or LUN went offline. In addition, the storage system performs background RAID scrubs to ensure that the data parity is correct and has not been damaged, corrupted, or is otherwise unusable. So, unless you lost 2 physical drives in a single RAID group (and an aggregate may have multiple RAID groups, depending upon the size of the aggregate), you wouldn’t lose the original LUN. I suppose that if someone were to try hard enough, we could delete the original snapshot upon which the FlexClones were based, but that would require privileged access to the storage system itself. In that case, I’d say you have bigger problems anyway.
I hope this makes sense. Thanks for commenting!
-
Scott,
Forgive my ignorance, I’m’ just trying to get my head round how this would work. It appears the majority of my conclusions were correct, but the snap applies to the volume and not the LUN.
Read / write snaps are nothing new really, neither are space efficient snaps so I’m applying my understanding of these to a flexclone, although I realise there’s no copy on write penalty for wafl.
What I don’t understand is how data blocks will not be shared if only for unchanged data, if the data remains unchanged post snap (frozen) in the parent volume then surely everyone accesses that same frozen data block ? Data’s data, it either consumes additional space or it’s shared as a single instance via a pointer.
If the data block does change then the clones will begin to diverge and consume their own storage space outside of the parent LUN’s blocks?
How easy would it be to predict and manage the additional space consumption given the storage is effectively oversubscribed ? I’m assuming this could be easily monitored to give plenty of warning.Cheers
John -
John,
You are correct; unchanged data blocks are shared between the original volume and any FlexClones created from Snapshots of that volume. As the original data blocks change or as data blocks in the FlexClones change, then the amount of shared data blocks will decrease and additional storage will be required (in this case on a NetApp system, the space would come out of the aggregate, unless the volume was created with a volume space guarantee).
The usage of space by the FlexClone(s) can be tracked via a number of methods (the command line interface “aggr show_space” or “df” commands, the FilerView GUI, and others). In addition, Data ONTAP 7.2 includes some new features for managing the usage of space, such as the “snap autodelete” and “volume auto_grow” features.
NetApp has put a fair amount of work into explaining thin provisioning (which is essentially what we are talking about here); check out these technical documents:
http://www.netapp.com/library/tr/3563.pdf
http://www.netapp.com/library/tr/3483.pdfI hope this helps. Please let me know if you have any other questions!
-
“As for contending for disk resources, the FlexClone will contend with other workloads on the storage system for the disks in the aggregate”
Hi, so with ONTAP 7.2 we offer FlexShare (no license required - priority command) where you can set execution priority for each volume relative to other volumes as well relative to system resources (i.e SnapMirror, SnapVault).
-
Nick,
Just curious–how would you envision FlexShare being beneficial in an environment where we are provisioning VMs using FlexClones? We can consider using FlexClones with LUNs (FCP or iSCSI) or using FlexClones with NFS.
-
Sorry for the delayed response Scott. In fact I just saw your reply.
Given that FlexShare is not tied to a specific protocol and applies on a per volume basis regardless of the type of the volume (Flexclone or not) and its contents (regular files or LUNs), there’s no protocol preference because it operates in the same manner in all cases.
In a multiprotocol array the ability to provide the peacful co-existence or various workloads regadless of the protocol is important. Afterall, you probably don’t want a volume with home dirs to have the same execution priority as a mission critical app. in an array appoaching full resource utilization.
In fact, I would equate FlexShare to VMware’s Resource Management where you can cap CPU and Memory resouces on a per VM basis. Now, if you take Flexshare and Multistore (which provides the ability to partition a physical filer into several virtual filers), what you end up with is capabilities similar to what ESX server is providing but for the storage side.
An area where Flexshare can be beneficial with Flexclone is Test/Dev. after you clone the primary volume, the clone resides on the same aggregate. So you may want to lower the execution priority of the flexclone in relation to the other volumes using the same disk resources.
Another area Flexclone can be beneficial is a case where one decides to split the Flexclone from the snapshot and create a fully autonomous volume. When that happens blocks are copied in the background from the snapshot into the Flexclone’s active space. Depending on the number of disks and back-end loops, this process can be quite intense (north of 350-400 MBps internal reads/writes). So you may want use Flexshare to limit the intensity of the back-ground copy.
happy holidays to you and your readers.
-
Nick,
Thanks for the response! I appreciate your time.
So the key to understanding FlexShare in this instance (when it applies to FlexClones for provisioning VMs) is that all volumes are considered equal, FlexClone or not. I did not know that a FlexClone would be considered the equivalent of a volume; with that information, it makes a lot of sense.
FlexShare, A-SIS, FlexClones, Snapshots–all these technologies make NetApp a really compelling storage solution for VMware environments.
Thanks again for taking the time to comment!
-
Hi Scott,
Yes Flexclones are considered equal to a typical volume. Anything you can do with a non-cloned volume can be done with a Flexclone. when you create a Flexclone from a snapshot, you’re actually creating a brand new volume that happens to share some blocks in the snapshot with the “parent” (original) volume. Writes to the Flexclone get allocated new space. Reads can come from the snapshot or from newly written blocks. Overtime, as the number of common(shared) blocks between the “parent” volume and the flexclone diverge, you have the option to split the flexclone from the snapshot at which point the “shared” blocks get copied out of the snapshot and into the space of the flexclone.
You can do the same thing with LUNs and the lun clone command.
There are more cloning enchancements on the way that server virtualization users will find extremely usefull not only from a povisioning but also from a recovery standpoint.
-
I’m familiar with the idea of a FlexClone (and a LUN clone, which I also wrote about at http://blog.scottlowe.org/2007/05/21/lun-clones-vs-flexclones/); I just hadn’t considered that they would be treated as equivalent to a “standard” FlexVol with regards to FlexShare and other operations. Now that you’ve pointed that out, it seems obvious, and I’m a bit embarrassed that I didn’t realize it myself!
I’m looking forward to future cloning enhancements!
-
Hi Scott,
Why would you hit the limit of 256 LUNs? If its flexclone/netapp, surely its running on NFS, not iSCSI/VMFS .. in which case you would hit the NAS datastore limit of 32…?
Cheers
Alex -
Alex,
This article was initially written from a block storage perspective, i.e., using LUNs, but you are absolutely correct–if we were using NFS, we’d run into the limit of 32 NAS datastores per ESX server.
Good point!
-
Scott, if we were using iSCSI/block storage/VMFS, then how would Netapp be able to do Flexcloning?
I was under the impressoin that the Netapp must use NFS for flexcloning so it can manipulate the files on the NFS filesystem.
Or can Netapp read VMFS filesystems?
Or does it not need to and by some magic flexcloning works with both iSCSI/VMFS and NAS/NFS?
Cheers
-
Alex,
NetApp can clone LUNs, either using FlexClones (in which case the FlexVol and the LUNs within that FlexVol are cloned) or using LUN clones (in which case the LUN is cloned within the same FlexVol). You’ll then need to deal with VMFS resignaturing, but the process is otherwise very similar. NFS is not a pre-requisite for using FlexClones.
BTW, do a site search for ‘FlexClone’ and you’ll turn up a number of articles, including a comparison of FlexClones vs. LUN clones.
Hope this helps!
-
Excellent work Scott. I know it’s late, but my question revolves around updating the master volume; Do the flexclones see the updates?
Specificaly, if I create a master volume with a golden XP image, can I auto update the master and propogate all changes out to the children in one whack? Will this cause (or potentially cause) inconsistencies in the childrens’ registry settings, say if a key was previously changed in the child?
-C
-
Christopher,
Updating the master volume does not update the FlexCloned volumes. The only way to update the clones would be to destroy and respawn them again.
For the kind of functionality you need, VMware View is the answer. I’m looking forward (I hope) to the vStorage APIs being extended to allow the storage arrays to do the heavy lifting that VMware View now performs in software, but today that’s just wishful thinking.
-
bummer - I thought you were going to say that… So if the original blocks are modified in the update in the master, the old pre-modified blocks are then in the snapshot the flexclones are based off? Or, does each flexclone get bigger as a result? Where are their pointers pointing?
Seems like there should be a way - kind of like unionfs in linux.
http://www.filesystems.org/project-unionfs.html-C
-
Christopher,
As the base volume is modified, the original blocks in that volume remain locked by the snapshot upon which the FlexClones were created. Additional space must be allocated to store the changes to the base volume, but the FlexClones do not require any additional space until they change their blocks from the blocks in the base volume’s snapshot.
Hope this helps!
-
we need dynamic clones. clones whose underlying base is malleable and can propagate immediately. You’re saying VMware View does essentially this, but this whole VDI thing adds an enormous amount of complexity. So much so that I’m seeking simpler alternatives by trying to leverage the NetApp directly in some way.
Ideally it’s an object oriented environment, where the child ‘class’ has inherited much of itself from it’s singular (or multiple modular) parent(s).
A spawned virtual environment is instantiated instantly on access, and is simply an extension of the dynamic class(es) it’s based off. ‘It’ is simply the change it brings to it’s instance.
Perhaps this is what ‘View’ essentially is? I’m not all that versed in it actually.
-C
Site Sponsors
19 comments
Comments feed for this article
Trackback link: http://blog.scottlowe.org/2007/05/17/netapp-flexclones-with-vmware-part-2/trackback/