I can take absolutely zero credit for this idea; it came completely from this aticle by Nick Triantos. But the trick is so absolutely cool, so incredibly useful, and yet so obvious (once you read it, you’ll smack yourself in the head and say, “Why didn’t I think of that?â€) that I just had to say something about it.
The use of NFS is getting more and more attention (I blogged about it briefly a few days ago) as a primary storage technology for VMware deployments. Although NFS lacks the raw throughput of Fibre Channel, once you start loading up VMs in a datastore NFS begins to look more and more attractive. But performance is only part of the allure here, especially when using something like a Network Appliance storage system with its Snapshot functionality. (Yes, other vendors can do the same kinds of things. Substitute your favorite vendor or filesystem here, if you so desire. I would imagine you could do something similar with ZFS.)
The basic gist of the article (I do encourage you to go read it; I’ve already added it to my del.icio.us bookmarks) is to use NetApp Snapshots to gain access to VMware’s VMDK files (even while the VM is running), and Linux with the Linux-NTFS driver to mount virtual machine disk files over NFS for file-level backups of both Windows and Linux guest VMs. Now that’s something not even VCB can do (VCB file-level backups are limited to Windows guests). Pretty cool, if you ask me.
Tags: NAS, NetApp, Snapshots, Storage, Virtualization
11 comments
Comments feed for this article
Trackback link
http://blog.scottlowe.org/2007/09/27/nifty-nfs-vmware-trick/trackback/
Saturday, September 29, 2007 at 1:05 am
Greg Perry
A similar idea, and one that doesn’t require a NetApp, is to compile FUSE in the COS, take a snapshot to release the flat .vmdk file, then mount or export the filesytem for NTFS reading. Older versions of FUSE can be compiled in the COS without issue.
Saturday, September 29, 2007 at 7:07 am
slowe
Greg,
That’s also a good suggestion and, as you mentioned, doesn’t require a Network Appliance storage system. However, some shops may be leery of a) compiling unsupported software in the COS; and/or b) introducing the additional CPU load in the COS. Nevertheless, it’s still a good idea.
Personally, I think the idea of using a Sun Solaris box (maybe a Thumper?) with ZFS shared out via NFS would be a good way to go about it as well.
Saturday, September 29, 2007 at 11:24 am
Greg Perry
Scott,
ZFS shared out via NFS for VMFS storage?
Saturday, September 29, 2007 at 2:56 pm
slowe
Greg,
Sure–use ZFS as the underlying file system, then share out the ZFS pool via NFS for ESX Server to use as a datastore. You can create snapshots via ZFS, share out the snapshot, mount it via NFS from a Linux box with the NTFS driver, and do the same thing.
This link talks about a similar situation, although they aren’t using ESX Server:
http://www.sun.com/bigadmin/content/submitted/zfs_mac_os.x.jsp
Enjoy!
Sunday, September 30, 2007 at 9:23 am
Greg Perry
Revisiting NFS as a storage container for VMFS is a very interesting prospect, and one that gets the EMC folks pretty excited (in a bad way) if you mention it as a replacement for FC or iSCSI.
Are you aware of any case studies out there that compare performance between iSCSI and NFS for VMFS shared storage?
Sunday, September 30, 2007 at 9:30 am
slowe
Greg,
See this article and the comments:
http://blog.scottlowe.org/2007/09/21/nfs-for-vmware-storage/
That should get you started…
Sunday, October 7, 2007 at 8:34 pm
Ron Terren
Scott,
What would your recommendaation be for a small shop running windows 2003 vm’s on the internal drives of the esx server to backup the machines. I had setup a box running freenas with an iscsi “lun” which I then mounted as a datastore. I would shut down the machines and rsync them to the iscsi datastore, but it means the vm’s must stay off for hours while the rsync happens. There is no SAN or fibre, all we want is to have a copy of the vms on another hd in another place off of the esx server. Money is tight.
Thanks Ron
Sunday, October 7, 2007 at 8:51 pm
slowe
Ron,
Good to hear from you again!
The rsync solution is a good solution, but you need to have a way for the systems to stay up. I would suggest taking a look at some of the scripts that have been written to take snapshots of the VMs; this unlocks the base VMDK so that you can run rsync on it. All the changes to the VM will be written to a differencing disk (a REDO log, if you will), which you can then commit once the backup is done.
A Google search should point you in the right direction.
The VMBK solution might work for you as well, see here:
http://www.vmts.net/vmbk.htm
Good luck!
Monday, March 3, 2008 at 2:02 pm
krdoor
you can not put VMFS on a NAS (NFS).
You can save vmdks (files) on nfs, but for creating a VMFS data store you either need:
- local storage
- san storage via FC
- san storage via ISCSI
Monday, March 3, 2008 at 9:38 pm
Aaron Delp
krdoor - I’m sorry but you can. We have customers that are running ESX 3.5 on HP Blades and the VMFS datastores are all on NFS hosted on NetApp.
The no NFS rule used to be true, but not anymore.
I’m Scott knows the history better than I do…
Monday, March 3, 2008 at 10:30 pm
slowe
Well–technically–you don’t put VMFS on NFS, as NFS will just leverage whatever underlying file system is already in place. But that’s just splitting hairs and being overly critical of the wording. Regardless of how you’d like to look at it, we can leverage NFS to store virtual machines, and in many cases using NFS can provide a number of benefits over more “traditional” storage protocols like Fibre Channel or iSCSI.