vMotion Practicality

About a month ago I posted an article titled The vMotion Reality. In that article, I challenged the assertion that vMotion was a myth, not widely implemented in data centers today due to networking complexity and networking limitations.

In a recent comment to that article, a reader again mentions networking limitations—in this instance, the realistic limits of a large Layer 2 broadcast domain—as a gating factor for vMotion and limiting its “practical use” in today’s data centers. Based on the original article’s assertions and the arguments found in this comment, I’m beginning to believe that a lot of people have a basic misunderstanding of the Layer 2 adjacency requirements for vMotion and what that really means. In this post, I’m going to discuss vMotion’s Layer 2 requirements and how they interact (and don’t interact) with other VMware networking functionality. My hope is that this article will provide a greater understanding of this topic.

First, I’d like to use a diagram to help explain what we’re discussing here. In the diagram below, there is a single ESX/ESXi host with a management interface, a vMotion interface, and a couple of network interfaces dedicated to virtual machine (VM) traffic.

VLAN Behaviors with VMware ESX/ESXi

As you can see from this highly-simplified diagram, there are three basic types of network interfaces that ESX/ESXi uses: management interfaces, VMkernel interfaces, and VM networking interfaces. Each of them is configured separately and, in many configurations, quite differently.

For example, the management interface (in VMware ESX this is the Service Console interface, in VMware ESXi this is a separate instance of a VMkernel interface) is typically configured to connect to an access port with access to a single VLAN. In Cisco switch configurations, the interface configuration would look something like this:

interface GigabitEthernet 1/1
  switchport mode access
  switchport access vlan 27
  spanning-tree portfast

There might be other commands present, but these are the basic recommended settings. This makes the management interfaces on an ESX/ESXi host act, look, feel, and behave just exactly like the network interfaces on pretty much every other system in the data center. Although VMware HA/DRS cluster design considerations might lead you toward certain Layer 2 boundaries, there’s nothing stopping you from putting management interfaces from different VMware ESX/ESXi hosts in different Layer 3 VLANs and still conducting vMotion operations between them.

So, if it’s not the management interfaces that are gating the practicality of vMotion in today’s data centers, it must be the VM networking interfaces, right? Not exactly. Although the voices speaking up against vMotion in this online discussion often cite Layer 3 VLAN concerns as the primary problem with vMotion—stating, rightfully so, that the IP address of a migrated virtual machine cannot and should not change—these individuals are overlooking the recommended configuration for VM networking interfaces in a VMware ESX/ESXi environment.

Physical network interfaces in a VMware ESX/ESXi host that will be used for VM networking traffic are most commonly configured to act as 802.1Q VLAN trunks. For example, the Cisco switch configuration for a port connected to a network interface being used for VM networking traffic would look something like this:

interface GigabitEthernet1/10
  switchport trunk encapsulation dot1q
  switchport mode trunk
  switchport trunk allowed vlan 1-499
  switchport trunk native vlan 499
  spanning-tree portfast trunk

As before, some commands might be missing or some additional commands might be present, but this gives you the basic configuration. In this configuration, the VM networking interfaces are actually capable of supporting multiple VLANs simultaneously. (More information on the configuration of VLANs with VMware ESX/ESXi can be found in this aging but still very applicable article.)

In practical use what this means is that any given VMware ESX/ESXi host could have VMs running on it that exist on a completely different and separate VLAN than the management interface. In fact, any given VMware ESX/ESXi host might have multiple VMs running across multiple separate and distinct VLANs. And as long as the ESX/ESXi hosts are properly configured to support the same VLANs, you can easily vMotion VMs between ESX/ESXi hosts when those VMs are in different VLANs. So, the net effect is that ESX/ESXi can easily support multiple VLANs at the same time for VM networking traffic, and this means that vMotion’s practical use isn’t gated by some inherent limitation of the VM networking interfaces themselves.

Where in the world, then, does this Layer 2 adjacency thing keep coming from? If it’s not management interfaces (it isn’t), and it’s not VM networking interfaces (it isn’t), then what is it? It’s the VMkernel interface that is configured to support vMotion. In order to vMotion a VM from one ESX/ESXi host to another, each host’s vMotion-enabled VMkernel interface has to be in the same IP subnet (i.e., in the same Layer 2 VLAN or broadcast domain). Going back to Cisco switch configurations, a vMotion-enabled VMkernel port will be configured very much like a management interface:

interface GigabitEthernet 1/5
  switchport mode access
  switchport access vlan 37
  spanning-tree portfast

This means that the vMotion-enabled VMkernel port is just an access port (no 802.1Q trunking) in a single VLAN.

Is this really a limitation on the practical use of vMotion? Hardly. In my initial rebuttal of the claims against vMotion, I pointed out that because it is only the VMkernel interface that must share Layer 2 adjacency, all this really means is that a vMotion domain is limited to the number of vMotion-enabled VMkernel interfaces you can put into a single Layer 2 VLAN/broadcast domain. VMs are unaffected, as you saw earlier, as long as the VM networking interfaces are correctly configured to support the appropriate VLAN tags, and the management interfaces do not, generally speaking, have any significant impact.

Factoring in the consolidation ratio, as I did in my earlier post, and you’ll see that it’s possible to support very large numbers of VMs spread across as many different VLANs as you want with a small number of ESX/ESXi hosts. Consider 200 ESX/ESXi hosts—whose vMotion-enabled VMkernel interfaces would have to share a broadcast domain—with a consolidation ratio of 12:1. That’s 2,400 VMs that can be supported across as many different VLANs simultaneously as we care to configure. Do you want 400 of those VMs in one VLAN while 300 are in a different VLAN? No problem, you only need to configure the physical switches and the virtual switches appropriately.

Let’s summarize the key points here:

  1. Only the vMotion-enabled VMkernel interface needs to have Layer 2 adjacency within a data center.
  2. Neither management interfaces nor VM networking interfaces require Layer 2 adjacency.
  3. VM networking interfaces are easily capable of supporting multiple VLANs at the same time.
  4. When the VM networking interfaces on multiple ESX/ESXi hosts are identically configured, VMs on different VLANs can be migrated with no network disruption even though the ESX/ESXi hosts themselves might all be in the same VLAN.
  5. Consolidation ratios multiply the number of VMs that can be supported with Layer 2-adjacent vMotion interfaces.

Based on this information, I hope it’s clearer that vMotion is, in fact, quite practical for today’s data centers. Yes, there are design considerations that come into play, especially when it comes to long-distance vMotion (which requires stretched VLANs between multiple sites). However, this is still an emerging use case; the far broader use case is within a single data center. Within a single data center, all the key points I provided to you apply, and there is no practical restriction to leveraging vMotion.

If you still disagree (or even if you agree!), feel free to speak up in the comments. Courteous and professional comments (with full disclosure, where applicable) are always welcome.

Tags: , , , , ,

39 comments

  1. Andy Kitzke’s avatar

    vMotion works great, but having a network background before I got into VMware really helped in understanding how the whole process works. It’s also a best practice to really give vMotion it’s own physical Network interface. Not to say you can’t share, but giving it dedicated and isolated bandwidth helps in troubleshooting processes later.

    I’ve also found it works great to use a single switch for the Conole port and the vMotion port and to assign two Physical nics but use them in Active/standby. Then the console and the vMotion ports get dedicated bandwidth while also attaining redundancy in the case of failover. The more complex your design gets, the more you can expand it as well.

  2. Erik’s avatar

    I really don’t understand where this push back against vMotion comes from. In my experience, the only major reason my clients have for not using vMotion is the cost of the Enterprise license. The networking requirements of vMotion are simple and straightforward, and even if you limit yourself to a /24 subnet for the vMotion interfaces that’s still 255 ESX servers. And since it’s a non routed subnet you can choose a class A if you really want to.

  3. MkH’s avatar

    Hi Scott,

    Excellent Article. I have configured our ESX servers in the same way and have had no issues with regards to L2/L3 problems. If one does follow VMware good practices, one wouldn’t necessarily encounter issues. In fact is a very secure way of doing it, especialy for the vMotion interfaces.

    Cheers

  4. Chris Waltham’s avatar

    Short answer: Yes.

    Long answer: We have around 170 VMs spread along 13 hosts, and several _hundred_ VLANs in our infrastructure. Each vSphere host typically has 40 or 50 VLANs trunked to each vSwitch, and each host usually has two vSwitches. Management VLAN is the same across every vSphere host. Migrating from one host to another works flawlessly for us, so long as the VLANs are trunked to each host (which we accomplish & maintain consistency via host profiles.)

    I honestly don’t know why the proposition that vmotions are a myth is given any respect. If you have to do long-distance vmotions, then sure, you have to consider a couple of things (like expanding your VLANs.) But that is a fairly trivial thing to keep in mind.

  5. Martin Barry’s avatar

    Scott, I think the issues people have with layer 2 are mostly around the VLAN limitations of the networking equipment and not anything related to vMotion/VMware.

    Take your example, if you wanted any VM to be run on any host then 2400 VLANs need to span across enough access switches to connect the 200 physical servers. The spanning tree protocol you are running needs to support 2400 VLANs. The upstream router(s) and/or firewall(s) need to support 2400 VLANs.

    You may also run into congestion issues if intra-VLAN traffic is no longer contained at the access layer and must traverse the aggregation layer. The more physical hosts, the thinner it is possible to spread a VLANs worth of hosts and the potential problems grow.

    cheers
    Marty

  6. slowe’s avatar

    Martin,

    I don’t agree with the assumption that there would be a 1-to-1 VM-to-VLAN mapping. In other words, having 2400 VMs is not, in almost all cases, going to mean 2400 VLANs. More likely, it will be 200 or 300 VLANs. So, from that perspective, the only “limiting” factor is the one mentioned by Chris Waltham: keeping ESX/ESXi hosts consistently configured across the data center.

    As for network congestion, I do agree that the network design has to be appropriately configured and that the congestion points can certainly move within the network. However, I don’t see this as a limiting/gating factor that would prevent organizations from effectively deploying vMotion within their data centers.

  7. Martin Barry’s avatar

    Scott, you are right in that the number of VLANs to support that many VMs may be, and probably is, a lot lower. However, my issues still stand as a lot of the defaults, at least in my experience with Cisco (YMMV), are in the low hundreds of VLANs. Mostly these can be raised with simple variable changes or possibly larger configuration changes (moving from RPVST to MST, speaking from personal experience).

    As such they should not be seen as hard limits but more issues that need to be addressed at the planning stage if they are not to cause scaling issues later.

  8. Paul V (sysxperts)’s avatar

    I think it’s also worth noting that anyone limited by the Enterprise licensing costs(as mentioned in Erik’s comment) will benefit from future releases rumored to be on the horizon, and I’m guessing that this class of customers would never require more than 254 vMotion enabled vmkernel addresses.

  9. Sysadmin’s avatar

    To me, this discussion seems to be less of a technical one, and more of an issue with many VM specialists having weak understanding of networks, their design, requirements, and technologies. It’s so overblown specifically because it’s presented as an issue for larger organizations, and it is such large organizations where we typically find narrow specialists each working on their own one thing. Their VM people don’t understand how networks work, and their network people don’t know what VMs need. And of course they can’t educate each other, because they don’t understand each other.

    On the other hand, if you look at a small company with a handful of ESX hosts, you’re likely to find just a couple of guys, who do everything: network, storage, VMs, guest OS, etc. They will quickly recognize a red herring, but nobody is going to ask their opinion, since to the GigaOm reporter they would seem to have no relevant experience.

  10. David Hesse’s avatar

    Good point from sysadmin.

  11. Russell’s avatar

    I come from a background of being both a network guy and a systems guy. I make it a point to bridge the gap between the two since both seem to have fundamental misunderstandings of each other or more often than not fear change.

    For example, in an environment with say 500 VLANs, many (say 75%) might be for client endpoints and all of the x86 servers can be found in maybe 30 or so VLANs. The network guy will argue that VMware needs to support 500 VLANs when the actual truth might be well under 10% of that. Understanding that a client would access the server through the same plumbing it always has (it just so happens that the edge switch the VM plugs into is virtual) helps these discussions go a long way.

  12. Brian Johnson’s avatar

    4.1 increases the number of concurrent VMs that can be vMotion’ed over a 10Gb uplink from 2 to 8 and it also increases the maximum throughput from just over 2Gb to over 8Gb.

    We have been testing the vMotion enhancements along with the new Network I/O Control feature and we have seen significant performance improvements.

    The vMotion bandwidth increase is very useful when using 10Gb uplinks that have not been limited to only a few gigs. For those concerned with vMotion taking to much bandwidth they can now use Network I/O Control to limit the bandwidth and set priorities to various traffic types. A cool new feature but under normal workloads I don’t think we will see the need to chop up a 10Gb uplink. The question I always ask is since we are now doing switch-to-switch connections (physical to virtual switch), not server to switch anymore, why are we trying to implement QoS when we normally don’t do it between switches now?

    Two 10Gb uplinks without QoS features enabled seem to work for most ToR switches so what has changed?

    10 to 20 gigs of outbound bandwidth across two 10Gb uplinks with 19.2Gbs of VM to VM traffic seems like a lot more bandwidth then what multiple 1Gb could ever dream of providing.

    Use the new load-based teaming with wide open 10Gb pipes and we should have enough bandwidth and headroom for most installations out there and it is a much easier to setup then having to carve up the pipe with one tool and setup a bunch of QoS on the switch and OS with others.

    Keep it simple, monitor it and then start moving the knobs when you really know you need it.

  13. Justin’s avatar

    Wait a minute. I agree with what you’re saying in this post, so I went back to read the comment on the previous post in order to try to understand what the real arguement against vMotion is here. Jeremy undoubtedly knows what he is talking about, though the arguement that “if the VLANs are not extended from Server Rm A to Server Rm B, you can’t vMotion a VM between the two and have the IP remain the same” seem like a bit of a ‘well duh’ arguement. That’s not a knock against Jeremy by any means, I believe someone above hit the nail on the head with the comment about networking guys and VMware guys not understanding each other and not communicating properly. Did the two of you have that sidebar afterall to discuss the VLAN sizing and distribution limits which would limit vMotion within the datacenter (or between datacenters for that matter)? I’d be interested to read more about that.

    It seems that the key points here are his argument that ‘most datacenters aren’t set up that way’, and your agruement that this is all possible ‘with a properly configured network’. Both of which are valid, they are simply looking at the subject from different viewpoints. As for the original arguement that vMotion is not an operational reality today, has that guy not talked to anyone in the industry? Sure, there are sizing limitations to anything if you scale it out large enough. That doesn’t mean it’s not already being done in production enviromnets.

    As virtualization becomes more dominant in the industry, we will see a shift in datacenter design which will provide a networking environment more conducive to VMs and VM mobility.

  14. Russell’s avatar

    Yeah its a bit odd; it struck me as him trying to present a problem to some solution that may not yet exist.

    Virtualization is pretty disruptive technology and anyone looking to seriously deploy it is going to make the necessary adjustments to their networking infrastructure to get the most bang for the buck.

    Honestly the only inhibitor I really see causing a lot of my large customers any headaches is in the CMDB and contractual issues. The technology piece is already sorted out and working just fine.

  15. FillDeeUK’s avatar

    Wow. Some great input From both VM guys and Network guys. Enough to blow most peoples brains out…mine included.

    I feel that the major argument seems to stem from network links between different sites. Room A to Room B, Datacentre A to Datacentre B etc. It seems that the belief is that vMotion is dictating a “replace you routers between these sites with bridges or it won’t work” approach. That creates the huge L2 network and the problems that go with it. Fact is, most networking gear now has the inteligence to “bridge” vLans, so only the necessary traffic hogs the site links.

    No one (not VMWare anyway) said flatten your networks. They just said “make a link where needed”.

    Of course (to be Devil’s Advocate), add in your “network cloud” provider and you might be pushing it to extend the vlans. Guess it keeps us in a job.

  16. Martin W’s avatar

    Brian: The simple answer to your question regarding QoS is that it is useful for hosting providers. You need to be able to limit your customers to what they are buying, or you’ll be getting yourself into a lot of trouble when one customer is using all the resources.

    Now regarding this vMotion post I think it is indeed an answer to a question that doesn’t exist. Or rather, it is an answer to someone who doesn’t know what they’re doing. If you’re a “VM guy” you need to have some networking clue. If you dont, hire a consultant!
    Seriously though, go learn some networking if you don’t understand how a VLAN or vMotion works, its pretty straight forward after all. Or are you going to teach networking 101 here Scott?

  17. Steve Puluka’s avatar

    Scott, a nice job explaining the issues and scope of the considerations. I just have a few references for folks.

    I second Martin W.’s call for the need for networking 101 for vmware administrators. Check out this free on-line course created by Juniper.
    https://learningportal.juniper.net/juniper/user_activity_info.aspx?id=769

    This is a nice reference design guide for the Vmware data center. See pages 13 and following for the vmotion issues:
    http://www.juniper.net/us/en/local/pdf/implementation-guides/8010009-en.pdf

    And of course the High Availability section of the Vmware site nicely organizes what we need to know.
    http://www.vmware.com/solutions/business-continuity/highavailability.html

  18. Omer Ansari’s avatar

    Filldeeuk nailed one of the two issues. It’s the L2 extension, and and I would add, the local storage which has historically made vMotion infeasible outside a DC
    Disregarding Vmware’s SRM for a second, when u vMotion a VM it comes up on another ESX server with it’s complete identity, including it’s IP address [1] and expects the ESX server it’s getting re incarnated on to have the same LUN mappings for storage [2]
    Imagine if u attempt the above across in a different geographically located DC.

    The company’s network would need to intelligently be able to route traffic for this VM now to the 2nd DC, while still routing traffic to all the other IPs in the same network as the VM to the first DC. That’s problem #1.

    Problem 2 is what filldeeUk brought up. Extending layer2 across large distances can cause instability because of the legacy L2 bridging protocol (STP)’s inherent limitations (extremely chatty, incessant need to recalculate topology at the slightest change etc etc).
    Problem 3 is the fact that your VM is now alive and kicking in DC2 but it’s storage is all the back in DC1. U can imagine the issues with this. Latency throughput etc etc. (SRM, and a well engineered storage network can solve this problem by the way)

    SRM cannot solve problem 1 and 2, short of changing your servers IP address (which may be an elegant enough solution if your app doesn’t care about the IP)

    Cisco’s OTP technology working in conjunction with LISP solves problem 2 and 1 respectively. OTV, amongst other things, modifies the behavior of STP across long distance L2 domains and LISP pieces out the ID of the end host from that of the site and enabling more intelligence into the network to route more surgically.

    Omer (FD: I work for Cisco)

  19. slowe’s avatar

    Steve P., Martin W., perhaps I should do a Networking 101 here for VMware administrators! Looks like there is a fair amount of demand. And thanks for the Juniper links, Steve.

    Omer, my response was primarily targeted at the use of vMotion *within* the data center, not between multiple data centers. As I pointed out in one of the comments above, the discussion about the practicality of vMotion *between* data centers is another story, and one that is still evolving. For proof of that evolution, consider EMC VPLEX, which is designed to help address the storage-related challenge you mentioned (VM in one data center with its storage in another data center). I haven’t yet had the opportunity to delve into LISP, so I can’t comment on that—perhaps you’d like to help me understand? I’d certainly welcome your input.

  20. Paul D. Pindell’s avatar

    I wanted to weigh in here as I think there is a misconception concerning vMotion requirements from a network perspective.

    There are four network related requirements in order for vMotion to succeed. These are: (a) The L2 characteristics must be identical, (b) the ESX(i) hosts must be in the same “vSphere Datacenter”, (c) the Port Group names must be identical, and (d) the storage used by the destination ESX(i) host must be accessible to the origin ESX(i) host.

    Let me expand on each of these four items a bit. These requirements are the same for local vMotion within a single datacenter, Metro distance (~200 km) vMotion (such as a Cisco OTV, LISP, DCE or other L2 stretching solution) between two well-connected datacenters, or a Long distance vMotion solution, which utilizes WAN optimization technologies to overcome link speed, and latency issues between two geographically distant datacenters.

    The first requirement is that the L2 characteristics of the network(s) to which the VM is/are connected must be the same (i.e. the IP address space, netmask, and Gateway must be the same). This might seem obvious but often I see folks who want to vMotion a VM from one location to another and are surprised that the VM can no longer connect to anything if the networks aren’t configured the same. If the gateway device of the network to which the VM was connected in origin had 192.168.0.1 as its IP, but the gateway device to which the VM connects in destination has 192.168.0.254 as its IP, the VM will not be able to connect to anything outside of its L2 broadcast domain. Nothing magic here, it is just worth stating.

    A successful vMotion does not require that the origin and destination networks actually are in the same L2 broadcast domain. There is no requirement to stretch the L2 broadcast domain. It doesn’t hurt to stretch the L2 broadcast domain, but there are other time-tested solutions for addressing overlapping IP space issues. Omer is correct in identifying problem #1 as needing to intelligently direct network traffic to the vMotioned VM once it moves. This is a standard function of an Application Delivery Networking device.

    The second requirement is that the esxhost-origin and esxhost-destination must in the same “vSphere Datacenter” construct. This is not a physical datacenter, but the grouping of ESX(i) clusters that constitute the logical boundary for vMotion.

    The third requirement is that the virtual machine Port Group name(s), which the VM being vMotioned has defined on esxhost-origin, is/are exactly the same on esxhost-destination. (Notice that VLAN IDs were not mentioned. As long as the Port Groups are named identically the VM will have no problem recognizing the network to which its vNIC(s) are attached. For example, a VM that has a vNIC connected to the Port Group named “Web-Servers” at origin, could successful be vMotioned to destination as long as there was a Port Group named “Web-Servers”. This is true even if the Port Group in origin was defined with a VLAN tag of 100, and the Port Group in destination was defined with a VLAN tag of 200.

    The fourth requirement is that both ESX(i) servers each have access to shared storage, and esxhost-origin must also have access to the shared storage of esxhost-destination. (There is no requirement that the primary shared storage for esxhost-origin has to be the same primary storage for esxhost-destination.)

    Notice that there is no requirement for the vMotion VMkernel interfaces of the ESX(i) hosts to have what was termed Layer 2 adjacency. The vMotion VMkernel interfaces are not required to be on the same subnet, VLAN, nor on the same L2 broadcast domain. The vMotion traffic on VMkernel interfaces is routable. IP-based storage traffic on VMkernel interfaces is also routable. Thus there are no L2 adjacency requirements for vMotion to succeed.

    In short, if you can route the traffic from origin to destination and can do so faster than both the storage is changing, and the memory footprint of the VM is changing, then you can vMotion.

    Paul Pindell (Full disclosure: I work for F5 Networks on VMware related solutions)

  21. slowe’s avatar

    Paul,

    Great information; thanks for sharing it here with the readers. Also, thanks for your full disclosure—I appreciate it! I am curious about the statement that there is no L2 adjacency requirement for the actual vMotion interfaces; I’m going to have to go back and review the VMware documentation on that one. I know that it *works*, but the question is supportability. Do you have a specific document that supports your position?

  22. Jose Maria Gonzalez’s avatar

    Hi Scot,

    Great stuff, much appreciated.

    Not sure where I heard this but I am pretty sure it was in a VMware event last year.

    VMware published some data as to how many customer that bought VMware products were using VMware in production.

    If I recall well VMware said 60% so not sure where this push back against vMotion comes from, but I might be wrong.

    For what I have seen so far, vMotion was well developed in the environments I have seen until now. I believe that this percentage could be higher now .

    Thanks

    Jose Maria Gonzalez from El blog de Virtualizacion en Español

  23. Paul D. Pindell’s avatar

    Scott,

    I’ve searched for a requirement in the vSphere 4.0+ documentation that would speak to this L2 adjacency requirement directly, but have not found one. On pages 187 ff of the vSphere 4.0, 4.0 U1, admin guide the networking requirements are listed, and L2 adjacency is not mentioned. Also in the latest vSphere 4.1 Datacenter Administrator guide pages 211-12 the networking requirements are listed, and again there is no mention (for or against) of vMotion traffic being on the same L2 broadcast domain.

    There is a new note in the DC admin guide that I am going to have to track down the reasoning that informs it. “NOTE You cannot migrate virtual machines that are attached to a virtual intranet with vMotion, even if the destination host has a virtual intranet configured with the same network label.” (p. 212)

    The concept “virtual intranet” is only used in this sentence within the guide. I believe it refers to the situation where a VM is connected to a vSwitch which does not have an external vNIC associated with the vSwitch.

    Paul

  24. Ole Andre  Schistad’s avatar

    Logically I cannot see any reason why the vmkernel interfaces should need to be on the same subnet, as long as the infrastructure can support the load. Obviously, doing a vmotion through a slow firewall is not a very good idea, but there are many examples of routing devices which add no significant latency or bottlenecks – such as the hardware routing in a cisco switch.

    mind you, while I haven’t tested this recently I seem to recall that you get a warning (but not an error) if you vmotion between two hosts where the vmkernel vmotion port group is named differently, but this is probably more a “heads up” to the admin in case he is in fact trying to move a VM through a (relatively) slow layer 3 device by mistake.

    Also the whole “devices per vlan” discussion is pretty dumb imho. The problem with a large broadcast domain is mostly linked to the actual broadcast traffic cumulated by having a large number of “chatty” devices share a network. Since the vmkernel does not try to autodiscover anything, it only needs to send broadcasts when initiating an IP connection to a device whose MAC-address is not already known. This results in one or two “arp who-has” packets, and then is shuts back up again.

    Sticking with that thought, let’s assume that you configure a dedicated vlan for vmotion (which is a good practice in medium+ size installations anyhow). When do you get broadcasts? only whenever an ESX host is starting a vmotion and needs to map the target ESX’ vmotion IP to a MAC address. Even if that host fails to respond you only get one broadcast every second or so, until the vmotion times out. So what’s that? Maybe 30 broadcasts in as many seconds.

    So I would think that you can stick a VERY large number of ESX hosts on the same subnet without incurring any significant overhead due to broadcast traffic. Probably a more relevant limit would be how large the mac-address-table on your switches can grow, but these days we are probably talking at least 4 digits.

  25. doublef’s avatar

    very interesting article..and great discussion

  26. Tee’s avatar

    Hello, I am a vSpecialist with EMC, and just saw this article and thought I’d share some information that is hopefully useful.

    I received this feedback from the software developer at vmware who wrote VMotion, and it was in response to the question of whether or not to route VMotion traffic.

    “This requirement dates back to the ESX 2.x days. Since 3.0, the vmotion vmkernel nic has been able to route traffic and thus you could have vmotion nics on different subnets. And I know a few customers do this. I’m pretty sure it’s officially supported. I’m not sure if we have any official documentation on this though.”

    Just for clarification: what the developer sees as possible with the software and what tech support will support in the field are two different things. However, there appears to be nothing documenting routed VMotion traffic is unsupported.

    I would suggest test the network to see if routing VMotion traffic shows acceptable performance, and proceed from there with implementing routed VMotion traffic or not.

  27. Omer Ansari’s avatar

    @Scott.

    Apologies it took my over a month to get back to your query on OTV/LISP, and in a way I’m glad I waited.
    There is a detailed article on networkworld written by a peer in my team on this. The OTV/LISP topic is broached on page 5/6

    http://www.networkworld.com/news/tech/2010/090310-layer2-data-center-interconnect.html

  28. Max Ardica’s avatar

    Hi guys,

    can someone explain once for all why L2 adjacency is required between the VMkernel interfaces of the source and destination ESX server? vMotion is happening leveraging a TCP session (port 8000), so what is the real reason why this traffic should not be routed? Is it merely a support issue on the VMware side or is there more about it?

    Thanks in advance,
    -Max

  29. slowe’s avatar

    Max, the VMkernel ports themselves do not require L2 adjacency. Refer to Paul Pindell’s comment above—the documentation no longer includes L2 adjacency as a requirement for the VMkernel ports. L2 adjacency is required, of course, for the front-end VM traffic.

  30. vmjfk’s avatar

    It’s easy to believe that vMotion interfaces cannot be routed. Since the ESX/ESXi server only has one default gateway, and that gateway is (usually) used for the service console network, and the vMotion network is supposed to be separate, that leaves no gateway for the vMotion network, and therefor no routability. In practical fact, the vMotion network needs to be L2 adjacent _unless_ you can do some fiddling with static routes to make it routable. So it’s not a formal requirement, it’s a limitation of the design of the system that can be (clumsily) overcome.

  31. Melvin’s avatar

    I’m not a vmware person. Never configured it and never done
    a vMotion ever. I dont think I have ever even see it done, so
    pardon my ignorance. let me ask a stupid question…when one
    performs a vmotion of a vm from one ESX host to another, isnt one
    of the desired results to maintain the original IP address of the
    VM’s vNIC? If so, then how can it be that there are no L2 adjacency
    requirements whatsoever when performing a vMotion? So, if ESX host
    1 has a VM on it with an IP address of 10.1.1.1/32 on VLAN 10,
    which is mapped to subnet 10.1.1.0/24, how can I keep the same IP
    address if I move the VM to another ESX host that is connected to a
    separate switch that doesnt support VLAN 10? Once again, pardon my
    stupid question.

  32. Melvin’s avatar

    Sorry for my clumy writing. I meant the VM has an IP
    address of 10.1.1.1/24 and VLAN 10 maps to subnet
    10.1.1.0/24.

  33. Melvin’s avatar

    Hello? Is this blog still active?

  34. Dan Robinson’s avatar

    @Melvin – there are newer network technologies out there that allow an IP Subnet to exist in 2 different physical locations.

    Note that an IP Subnet is not the same as a VLAN. Subnets are L3 and VLANs are Layer 2.

    Thus the same subnet 10.1.1.0/24 could be available on 2 different “Broadcast Domains” which are L2 VLANs, one for each location.

  35. Dan Robinson’s avatar

    I stumbled across this blog post rather late, but wanted to add something that is now more relevant for vSphere 5.

    You might want to double the size of your IP Subnets used for vMotion if you are planning large vSphere clusters/Datacenters.

    The reason being that in vSphere 5 there is a trick you can use to have more than 1 (2 in most cases) vmnic/vmkernel port being used for vMotion at the same time. So no longer is vMotion using 1 vmnic and the other is idle waiting for the first to fail. This is true even when a single VM is being Migrated.

    But why would you need a bigger subnet?
    The reason is to make this trick work, each vmnic now gets its own IP Address rather than a single IP address that fails back and forth.
    So you effectively need 2x the IPs for the same number of hosts.

    The details:
    http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2007467

  36. vignesh’s avatar

    Being a beginner in vmware and ucs, I have small doubt.
    What is limitation of changing mac address on vmotion?
    Why do we create mac address pool in service profile?

  37. Omar Khaled’s avatar

    Can someone explain this?

    “By default, you cannot migrate virtual machines that are attached to a virtual intranet with vSphere vMotion, even if the destination host has a virtual intranet configured with the same network label.”

  38. slowe’s avatar

    Omar, this means you can’t vMotion a VM when it is attached to a vSwitch that has no uplinks.

Comments are now closed.