A Few Thoughts on Xen

I’ll start this blog posting with the disclaimer that I am not a Xen expert (at least, not yet). That being said, I had some questions, thoughts, rants, etc., that I wanted to get out of my system after a lengthy meeting with some Citrix XenServer folks today.

For any readers out there that are intimately familiar with the Xen hypervisor and, specifically, Citrix XenServer, feel free to provide factual, relevant technical information in the comments. Salespeople and marketing drones, spare us all the advertisting and public relations drivel.

First, the XenServer folks had a lot to say about how their 64-bit hypervisor was better than VMware’s hypervisor because VMware’s product was “only” a 32-bit hypervisor. My main response to that statement is this: what key benefit do you derive from being a 64-bit hypervisor? Access to more system RAM? No, VMware Infrastructure 3.5 supports 256GB of RAM, while XenServer 4.0.1 supports 128GB of RAM. Do you gain the ability to run 64-bit guests? Well, VMware’s products can run 64-bit guests as long as the Intel VT/AMD V extensions are present. (Before you say that Xen doesn’t need the extensions to run 64-bit guests, better stop and think about the requirements to run unmodified guests of any bitness under Xen.) The ability to provide more RAM to guests? No, VMware beats Xen on that one, too—64GB versus 32GB. Hmmm…I’m stumped. So what does having a 64-bit hypervisor get you, then?

My second question centers around how the Xen hypervisor is so small and slim and petite versus VMware’s bloated 2GB hybrid OS-hypervisor (their description, not mine). Of course, it’s easy to make that distinction when you exclude the Linux-based Xen dom0 but include the Linux-based ESX Server Service Console. If want to compare hypervisor to hypervisor, fine; compare Xen itself with VMkernel. But don’t compare the Xen hypervisor with the whole ESX Server product; you’re not making an accurate comparison.

Third, let’s talk about paravirtualization. The Citrix XenServer folks loved to talk about paravirtualization and how Xen uses it but VMware doesn’t. What they seemed to omit, however, is the context in which we discuss paravirtualization. Does Xen use paravirtualization? Certainly—with guests that support paravirtualization, which does not include Windows. And they use paravirtualized drivers, to help optimize performance of the guest OS. Does VMware use paravirtualization? Yes, with guests that support paravirtualization, and that does not include Windows. Oh, and they do have paravirtualized drivers to help with guest OS performance. Where’s the difference? Feel free to tell me I’m missing something here.

Forgive me if it seems like I’m bashing Xen; that is most definitely not my intention. In fact, I’d love to get to know Xen much better than I do now. Products such as XenServer and Virtual Iron—both based on the open source Xen hypervisor—have the potential to make a huge splash in the market, and I for one want to be prepared to support these solutions when the time is right. I will tell you this, though: the wrong way to get to guys like me is with misinformation and marketing. Talk honestly and openly about your product’s strengths and weaknesses, and let your product stand, or fall, on its own merits.

Tags: , , , ,


  1. Simon Crosby, CTO  XenSource’s avatar

    Yah, who can blame salesfolk for saying what they do?

    Memory: Our limit of 128GB/host ($40K of memory!) is simply a testing limt. The architecture scales directly to 4TB. Does anyone care? I’m not sure yet, but likely XenDesktop will exercise a lot of host memory in some implementations.

    Paravirtualization: Yes, VMware with the vmxnet driver supports what we’d call a PV driver. Look, over time, the basic component of virtualization aka hypervisor and some basic machinery, gets (a) equivalent and (b) commoditized in all offerings. I’d be the first to welcome VMware to Paravirtualization and to declare equivalency of function. Where we do have an edge, and you don’t have all the facts (not your fault at all, by the way) is that .. with some careful work one can really optimize the performance of legacy Windows on Xen’s PV hypervisor – which we’ve done. So we do a very good job for Windows virtualization. Not all Xen implementations do, by the way – we regard that as proprietary value add.

    The key differences: We have an open, pluggable storage architecture with no clusterfs, but support a range of storage back-ends. We like it, storage vendors like it, customers like it. We don’t have a centralized management console that purports to be the god of all things virtual, and therefore represents a single point of failure. We have a redundant management architecture that explicitly endorses 3rd party plugins, and we go to market with a strong ecosystem of ISV add-ons, rather than excluding them from the opportunity.

    We haven’t ever shipped a hotfix, and VMware ships more than one per week.

    Other than that, we offer better price performance, and a portfolio of products that extends the value prop of virtualization up the app delivery stack, and through the network itself. From an end to end perspective, we have more ‘stuff’ to make the user’s use case work than VMware does, and therefore have a very powerful set of offerings.
    That’s about it.

  2. Toni Verbeiren’s avatar

    Good points.

    I noticed, in your own writing that most of the things that are claimed to be better with Xen are relatively new in VMware. 2 years ago or so, the statements were probably still valid, with VI 3 (especially from 3.0.1) they are no longer…

    Maybe the Xen guys need to look at their competitors more often than once every 2 years?

  3. slowe’s avatar


    Thanks for taking the time to respond.

    - OK, your point about the 128GB “limit” as a supportability limit is a valid one. I can understand that. But that doesn’t address the underlying question–what value do you feel Xen gains by virtue of being a 64-bit hypervisor? Going 64-bit just for the sake of 64-bit (and being “twice as good as the competition”) doesn’t cut it. (And yes, I agree that putting 256GB of memory into a server today would be outrageously expensive and therefore probably isn’t even in the realm of possibility. My point in bringing up limits is simply to address the whole “my hypervisor is a 64-bit hypervisor” argument.)

    - Are these Xen optimizations for running legacy Windows part of the hypervisor itself? If so, wouldn’t they need to be released back to the community under the open source license? If not, and if you can share that information here, to what optimizations are you referring? It can’t just be PV drivers, because pretty much everyone out there is using PV drivers.

    - I do like the “replicated cluster” approach with XenCenter. I hope you are working on addressing the need to manually elect a master.

    - I haven’t had the opportunity to more closely examine your interaction with storage. Supposedly, 4.1 will bring supportability to FC multipathing–a SERIOUS deficiency, in my mind, to not support FC multipathing (and yes, I know it worked, but customers generally aren’t going to implement something if it isn’t supported by the vendor).

    - Let’s be fair about hotfixes, though, and point out that the majority of the hotfixes that are being shipped are for the Service Console (the equivalent of your Dom0). In that case, it would be accurate only to say that XenSource did a better job of stripping down your Dom0 to the bare essentials needed instead of shipping a larger and more generic Service Console.

    - Price is definitely in your favor, at least for now. It remains to be seen whether VMware will counter with lower prices as competition increases. Will Citrix be able to reduce their prices as well? Will price always be in your favor?

    - Yes, Citrix does have a compelling suite of products (Provisioning Server is VERY nice). Of course, my article was strictly focused on the virtualization offerings and the hypervisor in particular.

    Again, thanks for taking the time to respond.

  4. slowe’s avatar


    Personally, I think the entire experience was the result of a misunderstanding, but I could be mistaken. As you can see, I’m trying to be out in the open about acknowledging where Citrix may have an edge over VMware and vice versa. We’ll see how it plays out.

    Thanks for reading!

  5. Brent O.’s avatar

    “We haven’t ever shipped a hotfix….”

    I got a chuckle out of that one. There’s thousands of projects on Sourceforge that can claim the same thing, but that isn’t a selling point. That tells me the software doesn’t have wide adoption yet, and/or the development is understaffed.

    Nothing against Xen – I’m sure it’s a solid product – but that one argument always makes me laugh.

  6. Dan C’s avatar

    It seems to me that what Xen are trying to do it commendable. That is, making inroads to a market that is predominantly held by a much larger competitor. I use Xensource in parallel to VI3 and am often pleasantly surprised at how neat and transparent, especially in an OSS sense, Xen proves to be. I can see why their sales folk might get overexcited when paraphrasing from their product sheets.

    On the other hand, reading Simon’s comments reminds me that they have to be careful that this won’t damage their credibility. Xen is not mature enough to be bragging about hotfixes at this stage in the day. SPOF from a centralised management console seems like a red herring. And so on.

  7. slowe’s avatar

    Brent O,

    That one is very likely to come back and bite them later.

    Dan C,

    Let’s just keep in mind that Xen (the open source project) is not the same as XenSource (the commercial entity that is now part of Citrix). I applaud Xen for working to create a great hypervisor that is also open source. Citrix, though, is out to make money–not that there’s anything really wrong with that, mind you. Let’s just be sure to keep the motivations straight.

    Like I said above to Toni, I think this particular case was simply a misunderstanding or a miscommunication. If that’s the case, Citrix better work harder on getting their people up to speed on how to intelligently and honestly compare and contrast their products against the competitor’s products.

  8. Matt Darlington’s avatar

    OK…..as a “misguided”, “misinformed”, “overzealous”, and “underinformed” sales person I must respond to your comments. First and foremost what you have stated and or intimated in your comments about what we said is incorrect, misleading, out of context, and unprofessional. At the beginning of your comments you stated “Salespeople and marketing drones, spare us all the advertising and public relations drivel.” Yet that is all your comments offered.

    You position yourself as a server virtualization “expert”, but you aren’t. You are a VMWare expert/evangelist. Folks like you can concentrate on being defensive / protectionist OR you can choose to really understand the broader server virtualization market with an eye toward positioning the right solution (regardless of vendor) for each client and not simply with a goal of being able to further your own agenda. One becomes a self-fulfilling prophecy about becoming irrelevant and the other approach leads to leadership recognition and growth.

    [NOTE: At Matt’s request, I have removed the majority of his original comment. Matt apologized for his hasty reaction and I felt it was only fair to remove the rest of his original comment. –Scott]

  9. Anon’s avatar

    An interesting read by all accounts. This does seem to be the way and ethos within Citrix in terms of their replies to almost all their new products they have released over the years.

    This reminds me of other products brought out by Citrix in the past. No one dismisses that the product works, or does ‘something’ better. However often Citrix hit the market too late, and human nature is to compare things with what’s already known.

    Off the top of my head
    -Application virtualisation (Drawbacks compared to SoftGrid)
    -Netscaler sorry Citrix Access Gateway (Required admin rights to run client, only works on windows platform etc)
    -Edgesight (very poor release won’t go into details) however amazing product now !
    -VDI Broker v1

    All these products promised to deliver but failed when comparing what’s already on the market. Long term as they mature they will either take over the existing established products on the market or become relatively equal and then a price war/customers experience become paramount.

    I think Scott’s views are very similar in the market place of other Vmware consultants as well as experienced customers. Sales people find it very easy to switch from say selling Vmware to Xen. But a consultants view of the world is always different.
    As a final note, although it’s a small point please drop the patch debate. 3 years ago Vmware we telling customers that their ESX do not ‘really’ require patching. Back then it was true, however with the uptake in customers so has the patch requirement. I’d be amazed if XenSource in the future does not end up the same way as customer usage grows.

  10. hot carl’s avatar

    scott, you instigator =)

  11. slowe’s avatar

    Wow. That’s quite a response.

    First and foremost, allow me to apologize for personally offending you. While you and I may never come to any sort of agreement with regards to technology, it would be un-Christian of me if I did not extend an apology to you. This was never intended to be a personal attack against either you or Keith. My choice of words was a poor one. So, for offending you, I apologize. Please extend my apology to Keith as well.

    Now, on to technology issues. Matt, how long did you say you’d been in the x86 server virtualization space? About 6 to 8 weeks? Isn’t it possible that, as a relative newcomer to this industry, mistakes were made or information was communicated incorrectly? Isn’t it possible that you haven’t had the opportunity to get into VMware’s architecture as deep as you might like? Isn’t it possible that you might have some incorrect information or incorrect assumptions about VMware’s architecture?

    Everybody’s human. Heck, I’ve been in the x86 server virtualization industry for several years, and I’ll be the first to admit that I’m not perfect. I don’t know everything. I’ll be the first to admit that I haven’t had the opportunity to get as deep into Xen’s architecture as I’d like. I’ll be the first to admit that I might have been given some incorrect information.

    (Oh, and by the way, I don’t claim to be an expert. Try reading more of my site than just this one article and you’ll see that.)

    Having said that, in the interest of making sure that the customer was given ACCURATE and FACTUAL information, I spoke up when I felt like you were perhaps speaking incorrectly. If you felt like I was pushing VMware’s agenda, or being confrontational, it’s only because the customer currently knows and understands VMware. I was asked by the customer to join the meeting to help them understand the differences between VMware and XenServer. How else to ensure the customer has a solid understanding of your product than to compare and contrast with something they already know and understand? Silly me…my years as a technical trainer taught me nothing.

    Specifically, there were several things you mentioned in the meeting that I felt needed clarification:

    - Comparing XenServer with VMware Server: As has been stated on this site MANY times, it’s not a fair comparison to compare any hosted virtualization product with a bare metal virtualization product. It wouldn’t be fair to compare Microsoft Virtual Server with XenServer, either. They are different architectures designed to server different purposes. Pardon the pun, but it’s like comparing apples and oranges.

    - Paravirtualization: You totally confused the customer with your discussion of paravirtualization by appearing to tie XenServer’s support for PV kernels with the use of PV drivers. Virtually (no pun intended) every major virtualization solution out there now uses PV drivers–VMware, XenServer, Novell, and Microsoft are ones that jump to mind (Microsoft calls theirs “enlightenments,” as I understand it). This isn’t a differentiator to XenServer; this is par for the course. Don’t get into a discussion about how XenServer is so much better than VMware (or any other product) because it uses paravirtualization when a) both products support PV kernels for Linux; b) both products use PV drivers to improve performance; and c) neither product uses PV to support Windows. Or am I missing something here? See, I’m willing to admit that I might be wrong. Show me where I am.

    - Hypervisor vs. hypervisor plus Dom0/Parent Partition/Service Console: Again, the statements made about the size of Xen vs. the size of ESX Server were misleading. You appeared to intentionally exclude XenServer’s Dom0 and include ESX Server’s Service Console in your comparisons. Perhaps that was accidental; I don’t know. But I do know that it was misleading to the customer. Again, let’s make fair and equal comparisons.

    - My comment to Simon above regarding automatic election of a new master was not sarcastic. In fact, you’ll note that I stated I like the distributed cluster approach that XenCenter uses. It effectively addresses what I perceive to be a problem with VMware’s management architecture.

    - You are correct that you did bring up the supportability limit with regards to 128GB of RAM during yesterday’s meeting. I apologize for the oversight.

    - Cold migration between Intel and AMD: Moved a VM–cold migration, not live migration–from an AMD platform to an Intel platform today. Worked fine, as I expected–and stated yesterday–that it would. Thought you might want to know, since you told the customer this was impossible.

    - 64-bit guest support: I’m still checking to make sure my information is correct, but it’s my understanding that there is no emulation involved. You are absolutely correct that emulating 64-bit operations would be horribly, horribly slow. It is my understanding that Intel VT/AMD-V is used to trap privileged instructions, and nonprivileged instructions are executed directly on the underlying hardware (VMware does require 64-bit hardware support, of course, just as XenServer does.)

    Finally, I’d just like to state that simplicity does not equal elegance. Simple solutions can be elegant solutions, but being simple in and of itself does not mean it is automatically elegant. The right solution should be as simple as it can be, but no simpler.

    Matt, I am open to further discussions, if you are interested. You can deepen my knowledge of Xen and Citrix’s specific implementation of Xen, and perhaps I can help deepen your knowledge of VMware’s architecture.

  12. Shawn Bass’s avatar

    Wow… Someone’s pissed in the Citrix SE’s corn flakes. Seriously, Scott is a customer’s representative. Tact would be to post a response that states that you disagree with the way things were stated in the blog entry and you’d like to privately meet to discuss the details. Then when everyone mutually agrees Scott could post the details stating where each party was misinformed (since I’m sure there’s mistakes on both parts). To respond by a verbal attack only makes Citrix look unprofessional. For that, you succeeded.


  13. Anon’s avatar

    Sorry, but I have to join in and agree with Scott here.

    I was also interested to know how long both Matt and Simon, and the other Citrix staff around the world have been working with Xen. I’ve seen far too much of this miss-information yet Citrix themselves appear to be pointing at everyone else.

    I think that a good healthy debate is great, however I am concerned that these things get shared around.

    Having been in similar meetings before (I can actually remember one quite clearly, well before XenSource was purchased by Citrix). I do see that customers who rely on ‘trusted’ sources are sometimes fed incorrect information.

    However, Vmware are now doing the same with their product set. The amount of meetings I’ve been in with Vmware pointing out that their own VDM/P2V/LabManager products are better than the rest is silly. Again I’m not saying they are not, but there are other established brands.

    In short, I think everyone tends to sit one side of the fence. And for those of us that sit on the fence (Scott and myself for example) others find it hard to see or really believe us because of our statements sometimes.

  14. slowe’s avatar


    As a further gesture to show you that this was not intended to be a personal attack, and in an effort to show that I want this to an open, honest, and professional exchange of information, I have edited my earlier comments and removed any wording that could be construed as a personal attack.

    Oh, and regarding hotfixes–what about this? Can anyone clarify for me?


  15. Keith Adams’s avatar

    I’ve worked in VMware’s virtual machine monitor group since 2000. I’m part of the three-person team who initially brought up 64-bit VMs on AMD hardware before SVM in 2004, and I am the one person team who brought up 64-bit VMs on Intel hardware with VT in 2005.

    Our “hypervisor,” the vmkernel, is indeed 32-bit in ESX 3i. However, we make a distinction between our “monitor” and our “hypervisor” that is unfamiliar to Xen folks. Our “monitor” runs at CPL-0 on the bare metal, but is instanced; there is one monitor per VM. The monitor is what actually emulates the CPU, constructs shadow pagetables, programs VT/SVM hardware if appropriate, etc.

    Our monitor has been 64-bit (on capable hardware) since 2004. Since the CPU-facing part of the software is 64-bit, it greatly reduces the pressure on us to move the vmkernel to 64-bit. Our customers understand this. When Xen Source sales people try to sell Xen by saying “Xen == 64-bit == new hotness, VMware == 32-bit == old n’ busted”, our customers are right to make fun of them on their blogs, IMHO.

  16. William Bishop’s avatar

    Sorry if I make this worse…but while Scott’s statements were a bit divisive, he’s not really relaying an experience we ALL haven’t been through.

    I leaned over in a meeting and whispered to the Hitachi rep to stop smearing IBM just the other day. He used the easy ambush, unfortunately I know IBM’s kit pretty well, so I gave him the polite warning. He did not repeat the mistake, and we parted on good terms. If anyone is from HD, please keep in mind that MOST storage vendors have some type of PFA…It’s rude to trot out what is a basic block of most vendors repertoire as if you are the only one that does it….

    Most SE’s take the competition too seriously and end up annoying customers. I have dismissed more than one vendor because they couldn’t seem to curtail their constant “betterness” than a competitor. I lose patience quickly in such instances, because most of the time I know both products already, they’re simply there because it’s procedural.

    That said, I have used both, and I administer a rather large installation of ESX(and further, I am an experienced citrix tech to boot). You would do better to leave out the portions of the pitch he mentioned. He’s obviously more patient than some of us. I would have asked you why you were not comparing size with 3i(where the add on is removed), and when you threw your second(and for me last) strike stating you can’t cold migrate between chip vendors, I would have politely asked you to leave. You have to know BOTH systems before you start comparing them, or you end up with egg on your face, and a lost sale.

  17. Mike DiPetrillo’s avatar

    Thank you, Scott!! I’ve been meaning to blog about this 64-bit nonsense for a while. Like you said, who cares? Exactly how is it limiting VMware? VMware beats XenSource in every category of support (more host memory, more guest memory, more 64-bit guests supported, earlier support for 64-bit guests, etc). There is absolutely no reason for VMware to move their hypervisor to 64-bit since the monitor (as Keith illustrates) has been there for over 3 years.

    Oh, and Simon, you really need to stop with the “we haven’t shipped hotfixes” statement. Reading is fundamental: http://forums.xensource.com/ann.jspa?annID=14.

  18. Come On’s avatar

    Wow! Sounds familiar. I had a VMWare SE do the same thing in reverse Today. I’m sure they aren’t Talking up their product vs. Citrix without knowing all of the details.

  19. slowe’s avatar

    Come On,

    It’s happened to all of us, from all vendors, at one time or another. There’s a lot of hyperbole out there, from all sides, as vendor vie for the business. I’m just trying–unsuccessfully, it seems–to get down to the facts.

    Thanks for reading!

  20. William Bishop’s avatar

    There is an easy fix, that I discovered last year. It works most of the time, if you are a big enough shop to warrant their agreement.

    Have them send you an engineer, not a salesman, and not a sales engineer.

    One of our vendors, after a couple se meetings, I asked if they would just send me an engineer, and they agreed. After two meetings, I told them never to send a salesman again. I’ve been hooked on this procedure every since. Works a treat!

  21. Roger Klorese’s avatar

    One point: the size comparison is often dismissed as being (XenServer hypervisor) vs. (VMware vmkernel + Console OS) — and, since dom0 is not included, unfair.

    If that’s what we ever actually said, it would indeed be unfair.

    But that is NOT the comparison we make, and never has been.

    The actual comparison is that (XenServer hypervisor) is much smaller than (VMware vmkernel + VMM) — that is, that the code for all the special-case emulation makes it huge and requires an army to support and enhance it, while Xen’s reliance on paravirtualization and hardware assist takes most of that complexity out.

  22. Roger Klorese’s avatar

    Furthermore, the argument is not that VMware *can’t* use paravirtualization for Linux — if it’s finally shipping anywhere other than on the desktop, that is. It is that it must carry around all the code that allows it to run Linux without hardware assist and without paravirtualization.

    As for paravirtualized Windows — we’ve made it known that we plan to run enlightened Windows Server 2008 at some time. Obviously, we don’t do so now. Will VMware?

  23. slowe’s avatar


    Thanks for joining in the discussion.

    Indeed, the comparison to which I was witness did exclude the XenServer dom0, which is what I pointed out. Your comparison–Xen hypervisor vs. VMkernel + VMM–is indeed much more accurate, fair, and a valid comparison to use.

    Of course, the flip side of that is the fact that VMkernel + VMM can do things that Xen hypervisor can’t do. So some–perhaps not all–of the difference in size (and complexity) is justified.

    With regard to support for paravirtualized kernels, that functionality is shipping in ESX Server 3.5, unless I am mistaken.

    I can’t speak to VMware’s ability to run enlightened Windows Server 2008, so it sounds like you may have an edge there over VMware.

    Again, thank you for your honest and forthright discussion.

  24. slowe’s avatar

    A final note to readers: I’m closing comments on this article (I’ll leave trackbacks open, though). I’d rather not run the risk of this getting too personal.

    If you want to continue the discussion throughout the rest of the “blogosphere”, post articles on your own sites and just trackback to this one.

Comments are now closed.