This is more of a “thinking out loud” sort of post, so bear with me. It doesn’t necessarily have a point, other than to prompt readers to start thinking.
VMware has VMFS. Microsoft has CSV, or will in Windows Server 2008 R2. (Does XenServer have a clustered file system? I don’t know.) What would happen if all the major hypervisors had a clustered file system in common? How would that impact virtualization?
I’ve written before about how cloud computing needs standards in order to really see broad adoption. People like to point to stuff like OVF, and that’s great, but what about brass tacks kind of stuff like file systems? Would a clustered file system that worked equally well with Hyper-V, ESX, and XenServer make any difference? What sort of flexibility or interoperability might it bring?
And, for further thought, what benefit would there be for that same clustered file system to be accessible from the guest operating systems running in virtual machines on these various platforms?
I’d love to hear everyone’s thoughts.
Tags: SAN, Storage, Virtualization
-
I prefer using NFS and letting the NAS head manage the filesystem. Less moving parts in the hypervisor to break.
-
I guess XenServer _could_ use GFS?
-
I’d like to ask a naive question:
Is there any technical limitation that makes it so you can’t run whatever clustered file system under a VMware, Microsoft, Citrix hypervisor?
In the open source world, Xen, KVM, etc. you could simply install your hypervisor on top of a clustered file system.
At first thought it seems that cluster file system compatibility is putting the cart before the horse so to speak, in that without everything else being compatible, OVF and the like, then being able to mix and match file system cluster nodes isn’t even a real possibility.
One benefit might be to make the cloud more diverse, but would it be possible to get VMware, Microsoft, Citrix, etc. to be compatible at that level?
Could it simply be done without their cooperation? Meaning, could some third party decide to implement the compatibility and support it? Maybe something of this nature is already in the works…
-
Shared file systems within a single platform is the key to unlocking many of the advanced features each vendor boasts.
Shared file systems across platforms is quite a radical idea and the doors that it could open would be aboslutely amazing for heterogeneous environments. Think about cross platform compatibility, cold and hot migrations between VMware, Hyper-V, XenServer, etc. Microsoft and XenServer already have this to a preliminary degree but I can’t believe that relationship will last forever. There was no love involved in that marriage and I believe it was arranged soley to wipe out VMware.
With a lot of these technologies being proprietary and the competition being as hot as ever, obvoius roadblocks would be participation, cooperation, and support from each of the vendors. Who sets the design standards? Is it design by committee? I personally can’t imagine the players working together at this level without a natural/instinctive agenda to inhilate each other. This is not at all ideal when the customer’s datacenters will realize the negative impacts.
I don’t see the virtualization vendors cooperating on a feat such as this. Each of the vendors believes firmly in their value proposition and would love nothing else but to wipe out their competition. To collaborate I think would mean admitting to a stalemate (or defeat) in the hypervisor wars and none of the major players are ready to submit this early in the game.
-
What about NFS? If you create NFS exports to a MS CSV volume, both Xen and VMware can reach the clustered filesystem that way, while Microsoft will access the filesystem natively.
Offcourse, this isn’t supported, and might not even work at all, but the first step has been taken (unknowningly) by all the major hypervisor vendors…
-
Scott, interesting thoughts. I would see the actual cluster files system as driven by the hypervisor and part of the secret source that differentiates them. The file systems should be abstracted from whats underneath, thats why in VMware we can have Datastores on NFS, iSCSI or FC and the important bit, the VM does not care what it is.
But you are trying to move beyond this and saying well why just have SCSI out of the VM, we have this great clustered file system, lets let the guest use it and share between guests.
It would need to be a new standard and I bet it would be a lowest common denominator as each vendor tried to position their sauce as better. Just like OVF is a good integration point but everyone adapts from there.
I wonder if there some open source clustered file systems already out there. Sun certainly have Lustre, so you wonder if this is a hypervisor layer issue or service/OS issue. http://www.sun.com/software/products/lustre/index.xml Re-inventing the wheel and all that.
Rodos
-
I think it would make Virtualization that much better. I also would like to see a standard hard disk file format. In theory you wouldn’t have to “convert” an existing VM if the hard drive format was universal across all VM products.
-
All,
Great comments, thanks everyone! One clarification: I’m primarily thinking of a clustered file system at the hypervisor level, so VMware ESX, Hyper-V, XenServer (and Xen derivatives) would use this clustered file system for storing their guests’ virtual hard disks.
I do agree that it’s highly unlikely the vendors will cooperate on this, so let’s assume that this is coming from an independent third-party.
Keep the discussion flowing!
-
Interesting idea, not sure that the Hypervisors sharing a single file system would gain much, maybe slightly more efficient usage of storage space? – if you wanted to do this to make moving between hypervisors easier they would all need to use OVF or similar as 1st class citizens for VM storage..
Thats probably going to be an ask; and the diversity could well be good from a reliability/security stand point
You could get into an emulation/virtualization layer for the file system itself.. VMFS/CSV–>back-end-format-FS etc. (this is kind of what traditional storage arrays do via FC/iSCSI albeit down to a block-level).
My middle ground where we need to mix hypervisors in a platform is that the back-end storage array is already agnostic/standardised – using FC & Zoning, NFS or iSCSI etc. if you need to move VM’s between different hypervisor vendors (and even physical hardware) then products like PlateSpin Migrate provide a good “portability hub” and you can then move server instances around between physical hardware (P2P, P2V, V2P) and different hypervisors (V2V) as your needs dictate – the conversion has an overhead and downtime but is that a big issue for this sort of change? –
A vMotion equivalent between hypervisors would be cool but would come with a lot of caveats that would make it risky from an operational point of view -a clear demarkation of vendor support boundaries can sometimes be a good thing.
On your last point I think there should definitley be isolation between the VM and the underlying file system – breaching that boundary could open a lot of risk for data leakage.
-
Would be nice indeed to have a generic filesystem for the hypervisor. this would make moving from hypervisor A to B more easy. and with the Cloud coming up it would make sense to run multiple different hypervisors for SLA purposes etc.
Although the chance of this happening is really small in my opinion.
-
Scott
I think there’s no chance of this happening. Firstly it would require vendors to open their formats for competitor evaluation and/or agree a standard format. We’ve not seen that happen in the storage array area where nearly every vendor implements their technology in a different way. The same thing occurs with the fabric where each vendor matches the basic standards but diverges with their own feature set.
We shouldn’t be surprised by this; vendors need to offer some value-add, otherwise they reduce to offering commodity products, reduce/stop innovation and become irrelevant.
Now, although I don’t think it will happen, I do think we need it. There’s far too much multi-standard stuff in IT which wastes resource/time and effort. The balance is getting that level of standardisation correct, without stifling innovation and creativity and so advancing the technology.
-
On the further thought…. If you are talking about the guests being able to read the cluster file system as something like an RDM, I think that could really open some doors for cross platform applications to work better together.
However my concern with the guests understanding the same cluster file system that the hypervisor could open a security risk. The way it is now no guest can read the file system to see the other vmdks. If they understand the file system the hypervisors would have to greatly change their permissions on those files to keep one guest out of another’s data.
-
All,
Good points about the guests reading the same file system as the hosts; that would indeed present some unique security issues.
-
In my opinion one need look no further than VMware or Xen on NFS to see the benefits. No vendor tool lock-in for provisioning, backing up, cloning, or repairing vm images. “Easy” porting of VM disks from Xen to VMWare Server to VMWare ESX and back.
Now, if that were a cluster FS, thereby eliminating the risks of a single NFS data source it would be all the better. Problem is, there are quite a few of these animals out there already. GFS and OCFS come to mind immediately, and I think Lustre fits the model as well. They are typically complicated to set up, finicky, and used only in highly specialized environments.
The biggest chance of something like this happening, IMHO, is pNFS. I don’t know a lot about it, but as I understand it it basically adds cluster scalability to NFS. VMWare and Xen already talk NFS, so it’s a natural next step. There are third party NFS clients for Windows as well, so a hyper-v port wouldn’t be out of the question.
WRT the security issues… Yes, there will be some… but they are just “different”, not necessarily worse. NFS access controls do work, they are just different than lun masking and the like. Coming from a Unix background instead of a storage background, I find I am much more confident in my ability to limit access using NFS than iSCSI =p
I definitely think the benefits outweigh the disadvantages!
-
This is a great discussion. Thanks Scott for starting it. Historically, file systems have seem the least amount of development compared to any other technology. Not surprisingly, Linux has seen the most activity.
I’d suggest that the shortest path to success with this would be a file system that runs on Linux – maybe one of the currently available ones (GFS??) could be used and further developed to add whatever clustering functionality is needed specifically for hypervisors and their guests.
Finally the abstraction layer needed to support whichever hypervisor there is on top of it. This is probably where the hardest bits would be – requiring qualification for all the various release levels of all the parts that touch FS operations. This is where patches tend to get ugly.
-
They do, it’s called NFS
-
I personally like the idea, although we might differ on how it could or should be implemented.
I simply do not think the hypervisor should own such a thing. Why would it? Think about networks – as an example – a system hooks up into the IP network and uses the services provided. It does not own them or manage any aspect of it. It simply leverages the service so it can concentrate on on what it does.
Why should this be any differnt with file systems? It seems we stuck in a crossroads looking longingly back into the sunset of proprietary OS’s rather than looking ahead to the bright day break of applications riding unhindered on their storage and network.
I am a storage guy. I worked for DEC, EMC and now NetApp promoting storage solutions. Philosphically, I think we have to look to storage to provide enterprise file services (like the network provides IP services for example). I like to manage the data where it resides.
In fact, if you run with what is out there today, the problems are half way solved – now you just need cooperation on VM file formats… but I won’t be holding my breath for that.
If they do cooperate, then they can take advantage of tried and true filesystems and free up everyone from hardware choices that do nothing but provide expensive vendor lock in. They’d very likely drop the cost of developmetn way down – and hence raise revenue.
Just a thought.
This is great and relevant discussion. Thanks for starting it.
-
Hi All,
I have been researching this same topic over the last few weeks looking for an answer to our growing storage problem at work. We are a software development house, the multiple test environments that are required use about 15TB. The cost of iSCSI or FC storage is expensive when some VM’s may not be operation for extended periods. After listening to a senior Google engineer it became obvious parallel clustered file systems can be down very low costs with very high throughputs. Though all of my research is showing there is not many out there who have found how to combine the two.
My thoughts were to use something like the PVFS2 roll for a Rocks Cluster. But this doesn’t present a file system that ESX 3.5 can use for the VM guests. If the clustered file system could present a iSCSI target then at least I could point at that for the guest to use….
Does any one have any ideas on how some thing like Lustre or PVFS2 could be used to create a common shared storage for VMware esx? -
Trackback from Dave Graham's Weblog on Thursday, June 18, 2009 at 1:32 pm
-
Not sure if this would work as a way to get high performance shared storage for all the desirable hypervisor features. Logically speaking it should work, but I haven’t found much on the topic while searching.
Using RHEL/CentOS with KVM as the hypervisor and the shared storage between nodes would be a Lustre FS mount. Create all the VMs on the Lustre mount and use 10gb Ethernet or the fastest supported interface by the OS on each physical host for the connection into the Lustre network. Have the same connectivity on the Luster metadata and storage controllers and you would have a very robust solution with shared storage. All without having SAN hardware or software lock in.
Site Sponsors
20 comments
Comments feed for this article
Trackback link: http://blog.scottlowe.org/2009/02/10/what-if-hypervisors-shared-a-file-system/trackback/