vMotion Layer 2 Adjacency Requirement

The topic of vMotion, it’s practicality, and Layer 2 adjacency for vMotion has been a topic I’ve visited a few times over the last several months. The trend got kicked off with a post on vMotion reality, in which I attempted to debunk an article claiming vMotion was only a myth. The series continued with a discussion of the practicality of vMotion, where I again discussed the various Layer 2 requirements for vMotion.

In the vMotion practicality article, reader Paul Pindell, an employee of F5 Networks, discusses the networking requirements for vMotion. To quote from his comment:

Notice that there is no requirement for the vMotion VMkernel interfaces of the ESX(i) hosts to have what was termed Layer 2 adjacency. The vMotion VMkernel interfaces are not required to be on the same subnet, VLAN, nor on the same L2 broadcast domain. The vMotion traffic on VMkernel interfaces is routable. IP-based storage traffic on VMkernel interfaces is also routable. Thus there are no L2 adjacency requirements for vMotion to succeed.

I was intrigued by this statement, so I contacted Duncan Epping (of Yellow Bricks fame) and discussed the matter with him. Duncan has also posted on this topic on his site as well; both his post and my post are the result of our discussion and collaboration around this matter.

So is Layer 2 adjacency for vMotion a requirement, or not? In the end, the answer is that Layer 2 adjacency for VMkernel interfaces configured for vMotion is not required; vMotion over a Layer 3 interface will work. The caveat is that routed vMotion, as it has sometimes been called, hasn’t been through the extensive VMware QA process and therefore is not yet supported. (Please don’t mistake my use of the word “yet” as any sort of commitment by VMware to actually support it.)

In summary, then: vMotion across two different subnets will, in fact, work, but it’s not yet supported by VMware. As additional information becomes available—as Duncan indicated, the VMware KB article is going to be updated to avoid misunderstanding—I’ll update this post accordingly.

Tags: , , , ,

  1. Roggy’s avatar

    For those out there considering this remember that the minimum bandwidth is I believe 622Mbps and RTT cannot exceed 5ms

    http://www.vmworld.com/docs/DOC-3840

  2. slowe’s avatar

    Roggy, you are correct. However, the consideration for using routed vMotion applies both to long-distance scenarios (where the requirements you mention are applicable) as well as data center-level scenarios (where bandwidth and latency are generally not an issue). Thanks!

  3. Paul D. Pindell’s avatar

    Scott,

    Thanks for digging into this deeper with Duncan.

    To throw a bit more fuel on the fire, I’ve been told by VMware employees that the 622Mbps and <5ms RTT requirements are not so much requirements for vMotion or Storage vMotion per se. The bandwitdth and quality requirements are more related to DRS, HA, etc, and VMware is officially not wanting to separate the vMotion requirements from the DRS and HS requirements. Thus the official supported requirement for vMotion is still 622Mbps and 5ms RTT.

    Paul Pindell

  4. Brandon’s avatar

    You and Duncan are connected at the hip I see :). As stated over on his post’s comment section — you are limited though to only 1 routable vmkernel interface. So if like in esxi, you choose to have your esxi hosts on separate networks, then your vmotion traffic will be limited to layer 2 adjacency as there is no way to specify more than one gateway for vmkernel. ESX’s service console is different in that you can specify a gateway for it, thus being on its own network, and then you could also potentially create a vmkernel interface with a totally separate network that is also routable. ESXi’s management interface is vmkernel, so the situation is different indeed.

  5. Dustin Pike’s avatar

    IF VMware were to support this, it would minimize the need for networking equipment to stretch Layer 2 traffic across the WAN for datacenter to datacenter vmotions. Has anyone played with this in a lab environment to see how well it works?

  6. slowe’s avatar

    Brandon, I’ve not tested this, but since the VMkernel is capable of routing traffic, I wonder if you could use esxcfg-route to support both a routed Management connection as well as a routed vMotion connection? Did you test this during your recent set of experiments with ESXi?

    Dustin, you would still need to make sure that the VMs have IP connectivity in both the source and destination networks. While there are a few different ways of accomplishing this, the most common is to stretch the VLANs between data centers so that the same VLANs are present in the source and destination networks. This ensures that there is no loss of VM connectivity after the vMotion operation.

  7. Steve S.’s avatar

    One thing to keep in mind, even though you may be able to successfully VMotion across a layer 3 boundary, the IP address of the guest OS will remain the same. The “layer 2 adjacency” requirement still applies for ESX hosts, just not necessarily for the vmkernel interface dedicated for VMotion. Unfortunately, the sprawling layer 2 data center architectures of today will continue to grow.

    It would be interesting to hear if any folks have deployed anycast services between data centers with a routed VMotion interface and without stretching layer 2 domains. Heavy management overhead but would be cool, none the less.

  8. Denise’s avatar

    Are you talking about a cold transfer, where you shut down a server, move it, and re-address it? Or are you saying that you can move a running server across a layer 3 boundary and have everything keep working (assuming IP connectivity is there without changing the server IP address.)

  9. slowe’s avatar

    Steve S, perhaps it’s just semantics, but there isn’t a Layer 2 adjacency requirement for the ESX/ESXi hosts. The Layer 2 adjacency requirement is for the virtual machines, not the ESX/ESXi hosts, and technically vMotion doesn’t really care if there really is Layer 2 adjacency. It will happily migrate a VM to a matching port group on the other end, regardless of whether that port group is actually configured to work. Like I said, perhaps it’s semantics, but sometimes the details are important.

    Denise, we’re talking about vMotion (live migration). What we’re saying is that there is a Layer 2 adjacency requirement for the VMs themselves (strictly speaking, they just need to have IP connectivity because the IP address doesn’t change during a live migration), but there is not a requirement that the vMotion network interfaces that are used by the ESX/ESXi hosts to perform the migration actually share a Layer 2 broadcast domain. I hope that helps!

  10. Paul D. Pindell’s avatar

    @Dustin You are correct that one would not need L2 stretching technologies to do a vMotion across the WAN. Duncan Epping, in his companion blog post to this one shares a link for Deploying BIG-IP to enable LD vMotion. This guide describes how to do just that. BTW, LD stands for Long Distance.

    @Steve S. As discussed in Scott’s earlier blog post even the VMs that are being vMotioned do not require L2 “adjacency” (excuse me for creating another term) but rather L2 “equivalency.” The L2 broadcast domains that the vMotioned VM had in the origin location have to be equivalent to the L2 broadcast domain in the destination location. One can achieve this with “stretching” the L2 across the origin and destination locations. One can also achieve this by defining two L2 broadcast domains with same network, mask, and gateway in each location. One could then connect those two identical L2 broadcast domains with a proxy or NAT device. LabManager uses “fencing” to do this.

    Paul Pindell (Fuller disclosure: I do in fact work for F5 Networks on our VMware Alliance team, but am not in this forum acting as an F5 spokesperson, nor heaven forbid a VMware spokesperson :-) )

  11. RussellCorey’s avatar

    Even if you route VMotion traffic itself you’re still going to want the network interface of the VM to land in the same broadcast domain. VMotion traffic itself may not require layer 2 adjacency but if your VM doesn’t end up in the right place its going to drop off the network.

  12. slowe’s avatar

    Russell, you’re correct of course. I actually like Paul’s term: Layer 2 equivalency. If you want your VM to continue to be accessible after a vMotion operation, you need to make sure that IP connectivity is maintained. Stretching L2 VLANs is one way to do that. Is it the easiest? I guess that depends on who you ask. :-)

  13. Tom’s avatar

    The relocation of a virtual server between two locations that don’t share a Layer 2 adjacency is difficult because of two things,
    1) the address resolution of the server from the incoming gateway ‘router’ typically requires ARP to function (which requires a broadcast medium).
    2) higher level routing protocols ‘eg ospf’ require information about the current location of the server. If they don’t have that information the traffic would be sent to the old location.

    Anycast isn’t really a suitable mechanism for what your wanting. Its designed for closest match routing. And in a corporate environment there is always a strong possibility of per packet load sharing (causing lots of difficulties) for most connection orientated services.

    Reverse Route Injection could be potentially used to achieve the aim of Steve S. For example you vmotion a windows IIS server from one site to another, it has loopbacks/secondary address’s associated with IIS. On the windows box you use RIP or any of the RAS software to communicate with the local router to indicate that the websites are at a different location.

    It would be an interesting concept, i’ve seen all of the above in use, but not mixed.

  14. Steve S.’s avatar

    @slowe @Paul When I hear the term L2 adjacency, I think sharing the same broadcast domain. Though it may not be a requirement on paper for ESX hosts, as you mentioned Scott, sharing the having L2 adjacency/sharing a broadcast domain/stretching VLANs, etc. are an implied requirement for a guest to be truly functional from a network perspective post-VMotion, without using anycast or a NAT device. The VMotion event may “work” but end-to-end connectivity with a guest will not.

  15. Brandon’s avatar

    @slowe — Duncan said you can use esxcfg-route over on his blog (in response to mine and one other’s posts), although I got the impression he thought I was dense for not realizing it myself :). I didn’t disagree with him as a static route is still layer 3, but that does add some additional complexity that seems rather unnecessary (meaning it should be configurable in the vSphere Client). You would have to remember and it still removes the benefit of using a gateway, but as I said over in his comments the traffic proably doesn’t change where it is going often enough to be a big deal. It’s also probably the very rare case you’d use anything but layer 2 anyway, but that isn’t the point of these posts. They really should add support for separate gateways if they want to support layer 3 vmkernel across the board.

    Another question I have about the above is… what if someone who was unfamiliar with the design came along and started looking things over. I can think of situations that might cause some confusion.

    I have not tested anything like this (so far), but my uses of esxi weren’t experiments as it is being used in production, but I suppose I can test this in my lab. Where I was implementing there was no requirement for routable vmkernel outside of the management interface. I was still able to tag the vmotion vmkernel port with a different VLAN than the management port which allowed me to create an unroutable vlan across the trunks.

  16. Sean Clark’s avatar

    Soooo, based on this it sounds like routing VMotion is okay. But for enabling VMotion over long distances, is routing VMotion preferred to running it over OTV (and routed underneath the covers)?

    My guess is that routing VMotion traffic would be preferrable since ESX servers aren’t mobile (yet) and since it seems like it would be the solution that would impose the least overhead on VMotion traffic over long distances.

  17. slowe’s avatar

    Sean, it looks like you won’t need OTV for the host-to-host vMotion traffic itself. Keep in mind, though, that routing vMotion isn’t yet fully supported by VMware.

    Aside from the host-to-host vMotion traffic, you’ll still need OTV (or its equivalent) to ensure that the VM still has connectivity after the vMotion operation is complete.

    Thanks for reading and commenting!

  18. vmjfk’s avatar

    Scott,
    Thank you for once again attempting to clear up the adjacency issue for vMotion. It’s my opinion that simplicity has inherent value. Not having static routes on individual VMhosts is benefit which outweighs the minor inconvenience of using something like OTV for the vMotion network, especially since you need OTV for the Vm network anyway.
    Kenny Lei and I have been doing some work recently with LISP (http://lisp.cisco.com) and OTV, which we demonstrated at EMCWorld 2011. LISP is in it’s beginning stages, to be sure, but is moving rapidly towards solving the L2 adjacency barrier issue and potentially minimizing the need for OTV. OTV or something like it is still required for nodes on the same subnet to communicate with each other after the vMotion, but nodes on other subnets will be properly routed by LISP.

  19. vmjfk’s avatar

    P.S. It looks like Beth Pariseau has updated her article in light of the comments you made above.

Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>