The vMotion Reality

The quite popular GigaOm site recently published an article titled “The VMotion Myth”, in which the author, Alex Benik, debunks the myth of vMotion and the live migration of virtual machines (VMs).

In his article, Benik states that the ability to dynamically move workloads around inside a single data center or between two data centers is, in his words, “far from an operational reality today”. While I’ll grant you that inter-data center vMotion isn’t the norm, vMotion within a data center is very much an operational reality of today. I believe that Benik’s article is based on some incorrect information and incomplete viewpoints, and I’d like to clear things up a bit.

To quote from Benik’s article:

Currently, moving a VM from a one physical machine to another has two important constraints. First, both machines must share the same storage back end, typically a Fibre Channel/iSCSI SAN or network-attached storage. Second, the physical machines must reside in the same VLAN or subnet. This means that inside a single data center, one can only move a VM across a relatively small number of physical machines. Not exactly what the marketing guys would have you believe.

Benik is correct in that shared storage is required in order to use vMotion. The use of shared storage in VMware environments is quite common for this very reason. Note also that shared storage is required in order to leverage other VMware-specific functionality such as VMware High Availability (HA), VMware Distributed Resource Scheduler (DRS), and VMware Fault Tolerance (FT).

On the second point—regarding VLAN requirements—Benik is not entirely correct. You see, by their very nature VMware ESX/ESXi hosts tend to have multiple network interfaces. The majority of these network interfaces, in particular those that carry the traffic to and from VMs, are what are called trunk ports. Trunk ports are special network interfaces that carry multiple VLANs at the same time. Because these network interfaces carry multiple VLANs simultaneously, VMware ESX/ESXi hosts can support VMs on multiple VLANs simultaneously. (For a more exhaustive discussion of VLANs with VMware ESX/ESXi, see this collection of networking articles I’ve written.)

But that’s only part of the picture. It is true that there is one specific type of interface on a VMware ESX/ESXi host that must reside in the same VLAN on all the hosts between which live migration is required. This port is a VMkernel interface that has been enabled for vMotion. Without sharing connectivity between VMkernel interfaces on the same VLAN, vMotion cannot take place. I suppose it is upon this that Benik bases his statement.

However, the requirement that a group of physical systems share a common VLAN on a single interface is hardly a limitation. First, let’s assume that you are using an 8:1 consolidation ratio; this is an extremely conservative ratio considering that most servers have 8 cores (thus resulting in a 1:1 VM-to-core ratio). Still, assuming an 8:1 consolidation ratio and a maximum of 253 VMware ESX/ESXi hosts on a single Class C IP subnet, that’s enough physical systems to drive over 2,000 VMs (2,024 VMs, to be precise). And remember that these 2,024 VMs can be distributed across any number of VLANs themselves because the network interfaces that carry their traffic are capable of supporting multiple VLANs simultaneously. This means that the networking group can continue to use VLANs for breaking up broadcast domains and segmenting traffic, just like they do in the physical world.

Bump the consolidation ratio up to 16:1 (still only 2:1 VM-to-core ratio on an 8 core server) and the number of VMs that can be supported with a single VLAN for vMotion is over 4,000. How is this a limitation again? Somehow I think that we’d need to pay more attention to CPU, RAM, and storage usage before being concerned about the fact that vMotion requires Layer 2 connectivity between hosts. And this isn’t even taking into consideration that organizations might want multiple vMotion domains!

Clearly, the “limitation” that a single interface on each physical host share a common VLAN isn’t a limitation at all. Yes, it is a design consideration to keep in mind. But I would hardly consider it a limitation and I definitely don’t think that it’s preventing customers from using vMotion in their data centers today. No, Mr. Benik, there’s no vMotion myth—only vMotion reality.

Tags: , , , ,

  1. Joe Onisick’s avatar

    Scott,

    Great post I think the thing missing from the assumptions in the “The VMotion Myth” is that people are willing to design their networks to maximixe vMotion capabilities in order to gain the plethora of benefits it provides. As you state the only L2 adjacency requirements are the kernel port of the hosts themselves. You can still build HUGE vMotion capable networks using tiny L2 domains.

    Joe

  2. Jason Boche’s avatar

    I don’t understand how anyone can knock the technical merits and utility of VMotion; it has proven itself useful and very reliable for years.

  3. Paul Richards’s avatar

    I was a bit puzzled after reading Alex’s post the first time but after reading it again, it seems that there are two main “problem” areas that he is pointing out. I use quotes around problem because I don’t necessarily believe that they are truly a problem. I see them more as challenges.

    The first is with server mobility within a single data center (and I read that to be a brick and mortar data center, not a vCenter data center). Scott, you are 100% correct on the requirements for vMotion: storage yes, network not so much (excluding VMK). And the part about moving VMs/workloads to a minimal number of physical machines (I’m running with the assumption that physical machines mean ESX hosts) is not based on inherent network limitations. You are free to migrate to as many ESX hosts as your network/vCenter will allow.

    The part about the service provider’s perspective really got my attention (since I work for one.) I agree that extending layer 2 to other physical locations can be somewhat “interesting.” I see the biggest challenges with this being complex infrastructure designs and the costs associated with them. That doesn’t mean solutions do not exist – take VPLS for example. Can you vMotion a system across a VPLS network? Probably. I think it really depends on a number of things like distance, circuit types/speed, and overall size of the VM. I could be wrong here…I’m by no means a network expert but those would be my primary concerns.

    I think everyone wants to see more progress on this subject regardless if they are an industry leader or a startup.

  4. Matt L’s avatar

    vMotion is more than just a functional reality, it’s an operational necessity. As manager of a relatively large environment for a span of years, moving out of esx version 2 all the way up through vSphere, vMotion was an enabling and mission critical piece of my infrastructure. Growing out of vMotion, the ability to dynamically migrate vm’s from host to host within a farm or cluster proved to be a lifesaver consistently. DRS, HA, and VMware update manager would not work at all were it not for vmotion.

    While it’s true that a shared infrastructure, similar processor stepping stones, a consistent networking schema, and consistent storage presented to all hosts within a cluster, these are simply the pieces of a well-designed and well maintained cluster. If your environment is properly configured, with discrete network vlan’s for service console, and vmotion, as well as trunked user segments, there is nothing that should stand in the way of vmotion.

    Storage vMotion is another piece of the puzzle. For a few years now, you’ve been able to migrate a VM from one lun to another, while maintaining complete uptime. This has only gotten easier. To move a vm from one lun to another, all that’s necessary is to have both luns presented to the same host. Should you choose to move the VM from one cluster to another, once again, all that’s necessary is that the storage be presented to bo9th clusters. It is recommended, though, that as the host will not reside in both clusters, you have the same consistencies in machine type and networking to support the vm’s configuration within both clusters. You may need to change the vlan to which the vm is attached, or reboot the vm if the guest will reside on a differing processor type.

    As to stretching your infrastructure over distance, there are now tools that will allow you to do this. I believe that Scott in a previous post, discussed the beauty of the vPlex infrastructure. This allows for a dynamic shared infrastructure across distances.

    Once again, reliance on a well architected infrastructure is all that’s required.

    Responding, once again, to Mr. Benik: vMotion is not a myth, but more than a reality, it’s truly a necessity. It’s a necessity, and it’s easy.

  5. B. Riley’s avatar

    Personally, I find it fascinating that anyone in the IT field would lodge a complaint like Benik’s. I mean, we’re talking about moving a machine off a physical host, and onto another without dropping any packets. Can anyone even imagine such a thing just a few years ago?

    The article is akin to complaining about the quality of peanuts on the Concorde.

  6. curtis’s avatar

    Thanks for this well thought out post! I read the GigaOm article this morning and my verbal response after finishing it was much less concise, used a series of expletives, and the details were certainly lost on my poor, ambushed wife. Even in the higher education sector, with our definite lack of funding, vMotion within the datacenter is the norm.

  7. Andrew Miller’s avatar

    Mildly disappointing that you’d even have to refute this….speaking as an architect/engineer who just implemented a full-on “VMware datacenter in a box” last week (storage + servers + vSphere), vMotion is definitely a reality even in shops with (4) ESX hosts — we had it up and going on day 2 actually (a good bit of day 1 was whiteboard time around VLANs and IP range layout….data center design really that should be done in any virtualized environment).

    The wonderful thing about vMotion (and DRS really, i.e. dynamic vMotion) is that you can stop thinking as much about individual physical hardware and just consider it to be a “pool of resources”….if you need more resources, add another physical box into the pool.

  8. Duncan’s avatar

    “While at some point that will undoubtedly be true, it’s far from an operational reality today.”

    Maybe he can do a site visit at one of our many enterprise customers doing fully automated DRS within their Virtualized Environment and thus moving workloads across multiple hosts etc. I have have customers doing it in a campus environment across sites.

    If you still think VMotion is a myth you are clearly living in the 90s.

  9. Dejan  Ilic’s avatar

    Actualy, I just thought about this matter the other day.
    I was wondering why Vmware hasn’t implemented vMotion to non-shared disks.

    With Storage vMotion, and now FT in place, VmWare do have most of the code to be able to do a vMotion over nonshared disks.

    True, it would take much more time to do the vMotion, but there might be circumstances when you would want to do this despite the time-factor. And it would open for cheaper systems with DAS instead, with ESX-host cluster pairs working off the same disk to provide the failure resilianse (easily possible with ie SCSI-boxes), again up to the designers wishes if it is a good solution. I can imagine the nightmare situation “I need to patch the ESX-host, please migrate off this host!”.

    And this should probably to a mostly “manual” procedure as certain disk-types doesn’t “support” storage vMotion ie RDMs. But these disktypes might not be a useable type in a virtualization envirovment with DAS-disks.

    Just my thoughts about the matter..

  10. Hari’s avatar

    Based on my (limited) knowledge, I agree with all the comments. But, I still see some practical limitations, in this context of VM migration. First, when used in conjunction with HA/DRS, the hosts will need to be in a cluster – from other erudite posts elsewhere, I understand that typical cluster sizes range from 4-8 hosts, not much more. In this case, even though Scott’s theoretical calculation is right on, in practice, you will still have limitations, right? Now, bring in CPU compatibility, despite datacenters being standardized with identical hardware, we are not there yet, fully.

    Just my 2c

  11. slowe’s avatar

    Hari,

    Cluster sizes are an important point, yes, but cluster boundaries are not vMotion boundaries. Administrators can still perform manual vMotion operations between separate clusters. Clusters only limit HA/DRS functionality to specific subsets of ESX/ESXi hosts.

    Thanks for commenting!

  12. Dewser’s avatar

    I am fairly new to vmware but I have read and witnessed enough to know that vMotion and Storage vMotion work. You don’t need a big fancy data center to see it in action. I read the article and my draw just dropped. Shoot just play the youtube video of DPM in action, that still blows my mind!!

    Thanks for clearing things up for him Scott!

  13. @that1guynick’s avatar

    I’d ask the writer of that article if he’s ever worked in ANY sort of clustered environment before, virtualized or not.

    Consistency (down to the processor stepping) and shared storage and complex VLANS are a NECESSITY, and not a limitation. We had to do that long before ESX was the “cool kid.”

  14. CanadianChris’s avatar

    The original article was silly. I suspect he was trolling for reaction. vMotion is a reality for many of vmware’s customers. Someone needs to either grow up or do their homework a little more thoroughly.

  15. Steve Karan’s avatar

    The biggest flaw in Benik’s blog is his complete lack of understanding of networking as a while (OSI model come to mind?). As a VCP for the past 4 years, to me, the foundation to a seamless and resilient virtual infrastructure is the network backbone (how else are end-users going to access the VMs). For this reason, the most important whitepaper from VMware would be the following:

    VMware ESX Server 3: 802.1Q VLAN Solutions
    http://www.vmware.com/pdf/esx3_vlan_wp.pdf

  16. Suttoi’s avatar

    Classic link baiting from GigaOm.
    Virtual is the default platform, and clustered is the default host config.
    vMotion will be on.
    HA will be on.
    DRS will be on.
    DPM & FT, Hmmm, sometimes.

    Om has been learning from his buddy, John C. Dvorak
    It’s John’s signature move.
    Say something contentious, then stand back and watch.

    Seems to be working too.
    I certainly read it.
    Although, now I come to think of it, I’m not sure why :-)

  17. Brian Johnson’s avatar

    I worked on a white paper with VMware showing how to setup vSphere 4.x using 10Gb Ethernet controllers.

    We did extensive testing of vMotion and we found that the it supported vMotion of VMs on different VLANs. One thing to note is all the host need to be connected to switches with VLAN trunking enable with ANY of the VLANs that VMs are using. This sounds obvious but we saw this issue when using 1G connection where multiple connections were used and the switch ports were not all configured the same. Since we only used two 10G connection we did not see the issue due to the reduction in switch ports that needed to be configured.

    Our testing also showed that when vMotion moved large memory VMs it did not take advantage of the increased bandwidth and by default only 2 VMs were able to be moved concurrently. Once VMware enable the full use of the bandwidth that 10Gb can provide.

    Here is a link to the Intel & VMware white paper.
    http://download.intel.com/support/network/sb/intelvmware10gbevsphere.pdf

  18. Dejan  Ilic’s avatar

    According to feature rumors för vSphere v4.1 it will support more concurrent vMotions concurrently.
    http://virtualization.info/en/news/2010/05/vsphere-4-1-features-leak.html

  19. Ashwin Jayaprakash’s avatar

    For engineers like me who rarely have direct exposure to datacenters, provisioning and that side of IT, would someone explain to us what kind of applications are hosted on these.

    I’ve listed down simple questions and described a pretty common deployment scenario in the Java/JEE world. I’d greatly appreciate answers (no flaming pls, we are noobs):

    1) The app running on the app server uses 80-90% CPU for 8 hrs a day
    2) The app is really a cluster of let’s say 12 processes running on 4 machines. App server already does FT/ HA and all that stuff. That’s why it’s a cluster
    3) Each process has been allocated and is using 3-4G of RAM
    4) Yes, these are JVMs
    5) 1G switch

    Given these basic stats,
    1) Would you put this setup on VMs? (Not to be confused with JVMs)
    2) Obviously this vmotion has to use CPU. Someone has to be syncing the deltas
    3) Again, it is also using the network and bandwidth
    4) Are you saying there is no performance impact on the application? No context switching, cache pollution, interruption, bus contention

    Or, is there a little elf sitting inside who can magically make this happen without impacting performance and guarantee 0 downtime?

    Explicitly clustered apps like App servers, Grids and NoSQL software and other such apps that people have been using for years, provide many knobs to suit the user’s combination of performance/FT/HA etc. They do not hide this – http://en.wikipedia.org/wiki/Fallacies_of_Distributed_Computing nor this – http://en.wikipedia.org/wiki/Leaky_abstraction from users/developers.

    Am I right or am I right? Perhaps someone can clarify such things for us? (without just asking us to RTFM pls)

    Thanks!

  20. Craig’s avatar

    Higher Bandwidth for vmkernel example 10G ethernet will cut short the time require to perform vmotion.

  21. Pradeep Padala’s avatar

    I have written a post explaining some inner details of live migration without all the gory details. http://ppadala.net/blog/2010/06/understanding-live-migration-of-virtual-machines/. Also, answering Ashiwn’s questions above. Hope this helps to some of your readers.

  22. Leif’s avatar

    In ESX 3.5 it was possible, though I’m sure not supported, to adjust the number of concurrent vmotions by modifying the vpxd.cfg file. I’ve never tested this in vSphere to see if it works still.
    http://www.boche.net/blog/index.php/2009/01/05/guest-blog-entry-vmotion-performance/

  23. Jeremy’s avatar

    All of the commentators here seem perhaps a little too quick to dismiss the GigaOM piece outright. While Benik’s article was certainly a bit vague at times, the underlying message that the *practical* use of VMotion to dynamically move virtual machines from host to host requires layer-2 adjacency (Common VLAN) between the targets is dead on.

    It is not the VMotion process itself which imposes this requirement, but rather the very assumption that a virtualized server can be moved from a physical host at one end of the data center to a different host in a completely different geography within the same (or different) facility and retain its network addressing as part of the process. While this can certainly be achieved in smaller data center environments, as the size of the facility(s) increases — the requirement of having many common VLANs which ‘span’ the entire site infrastructure to permit L2 mobility on-demand is generally not considered to be a scalable practice that promotes network stability.

    Most large enterprise data center environments must impose various L3 (routed) boundaries to break up L2 (switched) network environments into self-contained ‘blocks’ to meet scalability and fault isolation requirements demanded by most large enterprise organizations. L3 isolation within facilities may also be used to meet various security and regulatory requirements. These requirements are even more prevalent in service provider (‘cloud, etc’) environments.

    While VMotion itself can occur through L3 boundaries, the stark reality is that the the IP addressing the VM had in its former location on one side of a L3 boundary is NOT going to be valid once that host moves to its new location on the other ‘side’ of that boundary.

    The host can be re-addressed as part of the VM move, and DNS, etc can be updated to reflect its new location — but its far from the near ‘seamless’ move that can be achieved if the source and target were truly L2 adjacent. Network (and storage) architectures can certainly be tailored to allow L2 adjacency across large facilities in support of properly defined VM mobility requirements — however the common virtualization ‘sales pitch’ of any VM moving between any host in your facility (or remote facilities) any time you want — no problem” — is indeed quite a stretch. If you are in a *large enterprise* data center that allows complete L2 mobility between any hosts within the facility (i.e. every VM host anywhere in your facility can access any .1q VLAN via its data trunk interface(s) — then your network engineering team needs to start preparing their resumes, as they’re due for a major meltdown after provisioning such a large ‘flat-earth’ L2 model.

    Before anyone jumps on that last statement, yes, newer network switch virtualization technologies like Cisco’s VSS/VPC/OTV can certainly help you build a monstrous flat L2 network with significantly less risk. However, everything has its breaking point, and when you mitigate STP concerns with cross-chassis etherchannel, etc. then perhaps your next barrier to scalability will be with MAC learining rates, or MEC table limitations, etc.

    Violate the old mantra about building scalable and fault tolerant networks “route when you can, switch when you must” at your own (eventual) peril.

  24. slowe’s avatar

    Jeremy, you are still overlooking a key part of the configuration. The fact of the matter is that the Layer 2 adjacency is only a requirement FOR THE VMKERNEL PORTS. If you want to maintain separate L3 domains in different VLANs for different services, SLAs, security policies, etc., fine. The ESX/ESXi hosts can support multiple VLANs for VM access, so supporting ten different L3 VLANs on a pair of ESX/ESXi hosts is NOT A PROBLEM. The ONLY area where L2 adjacency is required is for the actual VMkernel ports where vMotion occurs. Thus, all these concerns about having to build massive L2 domains just to support vMotion is, in my opinion, rubbish. Continue to use L3 domains to separate VMs by department, business function, ownership, or whatever other metric you like; ESX/ESXi fully supports 802.1Q VLAN tagging and can talk to multiple VLANs at the same time. The single port that I have to configure for L2 adjacency per ESX/ESXi host—which might support 15, 20, 25, or 30 servers—is a small price to pay. The reality is that the practical use of vMotion in data centers today is fact.

  25. Jeremy’s avatar

    Scott,

    I am not talking about the actual requirement of L2 adjacency for the VM kernel ports. I agree with your position/comments — that particular requirement is of little practical concern.

    The point I was trying to get across (as was Benik in his article — I think) is that in order for a VM “Guest” to be *usable* when moved from one VM Host to another (via DRS, manually, whatever) — there is a presumption that the source ESX host and the destination ESX host will both be able to trunk the same VLANs from their local network switch.

    Example: If a web server (a VM guest) had a data NIC address of 192.168.1.1/24 in its current home (VM host) on one side of your datacenter — when it gets moved to a different VM host on the other end of your building — it still needs to retain that exact 192.168.1.1/24 address — thus both VM Hosts need access to the VLAN which holds that IP subnet.

    Of course this type of move could be dealt with in other ways, such as scripting server IP address changes and updating DNS, etc. as part of the DRS or manual VMotion move process — but that’s just not the ‘seamless’ mobility everyone’s been promised by the virtualization sales vendors.

    If you have a very large datacenter environment and VM is *everywhere* throughout it — there is really no practical way to have L2 ‘mobility’ throughout the entire facility enabling any VM guest to be moved to any VM host within any locale of the facility without running a ‘flat-earth’ (all L2 VLANs available anywhere) model that will undoubtedly get you into some serious trouble that increases the larger you become.

    Planning is certainly key here. If you know in advance that you want one L2 environment to support DRS between servers located in three different rows, etc. of your facility, then you design accordingly up front. If you’re able to do this and live with those confines, then great. If your working in shops small enough not to have to concern yourself with such things, then that’s wonderful too.

    Unfortunately, many large datacenters are not really all that well planned. VM sneaks its way in to the environment in random, unplanned ‘pockets’. Those pockets all suddenly ‘need’ to be part of the same farms/clusters Again, the expectation is out there (is often actively set by VMWare sales folks) that VMotion and DRS allows you to move any guest to any VM host in your facility like magic. Never mind the contortions the network is bent into on the back side to make that vision a reality. Never mind how unstable such an environment can silently become.

    Take the VM part away from this discussion for a second, and think of this in terms of physical servers. If it was cost-effective to physically move servers around the datacenter (which it of course isn’t) — can you think of many large datacenters where you could just move a server from one row to another, one building to another, one campus to another, etc. and get to just take your (L3) addresses with you without any special arrangements in advance?

    -Jeremy

  26. slowe’s avatar

    Jeremy,

    OK, thanks for the clarification. You appear to be speaking from first-hand experience, so help me understand: where is the tipping point here? At what point does the number of VLANs tip the scales too far? Or if it’s not the number of VLANs but instead the breadth of distribution of those VLANs, what is the breaking point? And how much of this is mitigated by proper network design (which, I’ll grant you, not every organization has the pleasure to do adequately)? I want to understand your position more fully, so any information you can share would be great. You’re welcome to e-mail me separately if you’d like, and I’ll post the results of our separate discussion in a future blog post.

  27. Melvin’s avatar

    Jeremy? Would love to hear more of your input.

  28. nicone’s avatar

    we just got into this whole thing trying to plan it all out

    have 2 geographically different data centers
    dedicated line between
    2 esx servers at each location
    each side will have its own san that replicates (every hour)

    just stuck on deciding if both sides should be on same subnet
    or use different subnets with scripting ip changes/dns updates

    having the same subnet on both ends really seems like the simplest approach

    but after reading these comments, having second thoughts

  29. Dennis Procopio’s avatar

    Can I perform svMotion across a layer 3, firewalled boundary? If so, what ports need to be open? My source datastore would be FC and my target would be NFS.

  30. southpaw’s avatar

    hello i have a networking question that might be elementary but i knwo from reading that someone on here can answer my question.

    1.how many and what layer 2 vlans are required to have a stretch cluster network across two datacenters. (vmotion, storage(iSCSI), storage(hp p4000))
    2. what vlans can reside only at each datacenter? (vmware esx hosts, ilo, active directory servers
    3. each datacenter will have its switches configures with the same production vlans and ip’s so vmotion’d server will functon properly when moved.
    4. i dont want to have excessive broadcast traffic if not needed.
    5. active directory can sync across L3 so that is why i dont need it to vmotion it can just do HA on each side.

    if someone could direct me to best practices drawing i would be gratefull.

    -southpaw

Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>