I can take absolutely zero credit for this idea; it came completely from this aticle by Nick Triantos. But the trick is so absolutely cool, so incredibly useful, and yet so obvious (once you read it, you’ll smack yourself in the head and say, “Why didn’t I think of that?â€) that I just had to say something about it.
The use of NFS is getting more and more attention (I blogged about it briefly a few days ago) as a primary storage technology for VMware deployments. Although NFS lacks the raw throughput of Fibre Channel, once you start loading up VMs in a datastore NFS begins to look more and more attractive. But performance is only part of the allure here, especially when using something like a Network Appliance storage system with its Snapshot functionality. (Yes, other vendors can do the same kinds of things. Substitute your favorite vendor or filesystem here, if you so desire. I would imagine you could do something similar with ZFS.)
The basic gist of the article (I do encourage you to go read it; I’ve already added it to my del.icio.us bookmarks) is to use NetApp Snapshots to gain access to VMware’s VMDK files (even while the VM is running), and Linux with the Linux-NTFS driver to mount virtual machine disk files over NFS for file-level backups of both Windows and Linux guest VMs. Now that’s something not even VCB can do (VCB file-level backups are limited to Windows guests). Pretty cool, if you ask me.
Tags: NAS, NetApp, Snapshots, Storage, Virtualization
-
A similar idea, and one that doesn’t require a NetApp, is to compile FUSE in the COS, take a snapshot to release the flat .vmdk file, then mount or export the filesytem for NTFS reading. Older versions of FUSE can be compiled in the COS without issue.
-
Greg,
That’s also a good suggestion and, as you mentioned, doesn’t require a Network Appliance storage system. However, some shops may be leery of a) compiling unsupported software in the COS; and/or b) introducing the additional CPU load in the COS. Nevertheless, it’s still a good idea.
Personally, I think the idea of using a Sun Solaris box (maybe a Thumper?) with ZFS shared out via NFS would be a good way to go about it as well.
-
Scott,
ZFS shared out via NFS for VMFS storage?
-
Greg,
Sure–use ZFS as the underlying file system, then share out the ZFS pool via NFS for ESX Server to use as a datastore. You can create snapshots via ZFS, share out the snapshot, mount it via NFS from a Linux box with the NTFS driver, and do the same thing.
This link talks about a similar situation, although they aren’t using ESX Server:
http://www.sun.com/bigadmin/content/submitted/zfs_mac_os.x.jsp
Enjoy!
-
Revisiting NFS as a storage container for VMFS is a very interesting prospect, and one that gets the EMC folks pretty excited (in a bad way) if you mention it as a replacement for FC or iSCSI.
Are you aware of any case studies out there that compare performance between iSCSI and NFS for VMFS shared storage?
-
Greg,
See this article and the comments:
http://blog.scottlowe.org/2007/09/21/nfs-for-vmware-storage/
That should get you started…
-
Scott,
What would your recommendaation be for a small shop running windows 2003 vm’s on the internal drives of the esx server to backup the machines. I had setup a box running freenas with an iscsi “lun” which I then mounted as a datastore. I would shut down the machines and rsync them to the iscsi datastore, but it means the vm’s must stay off for hours while the rsync happens. There is no SAN or fibre, all we want is to have a copy of the vms on another hd in another place off of the esx server. Money is tight.
Thanks Ron
-
Ron,
Good to hear from you again!
The rsync solution is a good solution, but you need to have a way for the systems to stay up. I would suggest taking a look at some of the scripts that have been written to take snapshots of the VMs; this unlocks the base VMDK so that you can run rsync on it. All the changes to the VM will be written to a differencing disk (a REDO log, if you will), which you can then commit once the backup is done.
A Google search should point you in the right direction.
The VMBK solution might work for you as well, see here:
Good luck!
-
you can not put VMFS on a NAS (NFS).
You can save vmdks (files) on nfs, but for creating a VMFS data store you either need:
- local storage
- san storage via FC
- san storage via ISCSI -
krdoor – I’m sorry but you can. We have customers that are running ESX 3.5 on HP Blades and the VMFS datastores are all on NFS hosted on NetApp.
The no NFS rule used to be true, but not anymore.
I’m Scott knows the history better than I do…
-
Well–technically–you don’t put VMFS on NFS, as NFS will just leverage whatever underlying file system is already in place. But that’s just splitting hairs and being overly critical of the wording. Regardless of how you’d like to look at it, we can leverage NFS to store virtual machines, and in many cases using NFS can provide a number of benefits over more “traditional” storage protocols like Fibre Channel or iSCSI.
-
Another way to look at the VMFS over NFS mis-understanding is to take a look at a Samba server. Samba serves the CIFS file sharing protol (Windows) but in reality the files are stored on a Unix filesystem (ext2/3, UFS, JFS, etc.) and not on an NTFS filesystem like a Windows server would store local files on a local disk/LUN (SCSI, SATA, SAS, FC, iSCSI, FCoE, and IDE partition).
The opposite analogy would be to say that Samba allows you to store NTFS on a Linux box via CIFS, but it doesn’t, it just allows you to put files on another storage system which will be using some type of Unix-like filesystem.
I believe the confusion easily arises as ESX stores the NFS mount points under the /vmfs directory so any volumes mounted here could be misunderstood as VMFS filesystems.
Site Sponsors
12 comments
Comments feed for this article
Trackback link: http://blog.scottlowe.org/2007/09/27/nifty-nfs-vmware-trick/trackback/