Comments on: vMotion Practicality

By: Melvin

Melvin — Sat, 01 Jan 2011 22:47:34 +0000

Hello? Is this blog still active?

By: Melvin

Melvin — Wed, 29 Dec 2010 21:35:48 +0000

Sorry for my clumy writing. I meant the VM has an IP
address of 10.1.1.1/24 and VLAN 10 maps to subnet
10.1.1.0/24.

By: Melvin

Melvin — Wed, 29 Dec 2010 21:02:30 +0000

I’m not a vmware person. Never configured it and never done
a vMotion ever. I dont think I have ever even see it done, so
pardon my ignorance. let me ask a stupid question…when one
performs a vmotion of a vm from one ESX host to another, isnt one
of the desired results to maintain the original IP address of the
VM’s vNIC? If so, then how can it be that there are no L2 adjacency
requirements whatsoever when performing a vMotion? So, if ESX host
1 has a VM on it with an IP address of 10.1.1.1/32 on VLAN 10,
which is mapped to subnet 10.1.1.0/24, how can I keep the same IP
address if I move the VM to another ESX host that is connected to a
separate switch that doesnt support VLAN 10? Once again, pardon my
stupid question.

By: vmjfk

vmjfk — Wed, 01 Dec 2010 23:04:19 +0000

It’s easy to believe that vMotion interfaces cannot be routed. Since the ESX/ESXi server only has one default gateway, and that gateway is (usually) used for the service console network, and the vMotion network is supposed to be separate, that leaves no gateway for the vMotion network, and therefor no routability. In practical fact, the vMotion network needs to be L2 adjacent _unless_ you can do some fiddling with static routes to make it routable. So it’s not a formal requirement, it’s a limitation of the design of the system that can be (clumsily) overcome.

By: slowe

slowe — Wed, 01 Dec 2010 22:59:52 +0000

Max, the VMkernel ports themselves do not require L2 adjacency. Refer to Paul Pindell’s comment above—the documentation no longer includes L2 adjacency as a requirement for the VMkernel ports. L2 adjacency is required, of course, for the front-end VM traffic.

By: Max Ardica

Max Ardica — Wed, 01 Dec 2010 16:57:53 +0000

Hi guys,

can someone explain once for all why L2 adjacency is required between the VMkernel interfaces of the source and destination ESX server? vMotion is happening leveraging a TCP session (port 8000), so what is the real reason why this traffic should not be routed? Is it merely a support issue on the VMware side or is there more about it?

Thanks in advance,
-Max

By: Omer Ansari

Omer Ansari — Tue, 07 Sep 2010 17:49:28 +0000

@Scott.

Apologies it took my over a month to get back to your query on OTV/LISP, and in a way I’m glad I waited.
There is a detailed article on networkworld written by a peer in my team on this. The OTV/LISP topic is broached on page 5/6

http://www.networkworld.com/news/tech/2010/090310-layer2-data-center-interconnect.html

By: Tee

Tee — Sat, 14 Aug 2010 15:39:57 +0000

Hello, I am a vSpecialist with EMC, and just saw this article and thought I’d share some information that is hopefully useful.

I received this feedback from the software developer at vmware who wrote VMotion, and it was in response to the question of whether or not to route VMotion traffic.

“This requirement dates back to the ESX 2.x days. Since 3.0, the vmotion vmkernel nic has been able to route traffic and thus you could have vmotion nics on different subnets. And I know a few customers do this. I’m pretty sure it’s officially supported. I’m not sure if we have any official documentation on this though.”

Just for clarification: what the developer sees as possible with the software and what tech support will support in the field are two different things. However, there appears to be nothing documenting routed VMotion traffic is unsupported.

I would suggest test the network to see if routing VMotion traffic shows acceptable performance, and proceed from there with implementing routed VMotion traffic or not.

By: doublef

doublef — Wed, 11 Aug 2010 17:34:36 +0000

very interesting article..and great discussion

By: Ole Andre Schistad

Ole Andre Schistad — Thu, 29 Jul 2010 13:05:08 +0000

Logically I cannot see any reason why the vmkernel interfaces should need to be on the same subnet, as long as the infrastructure can support the load. Obviously, doing a vmotion through a slow firewall is not a very good idea, but there are many examples of routing devices which add no significant latency or bottlenecks – such as the hardware routing in a cisco switch.

mind you, while I haven’t tested this recently I seem to recall that you get a warning (but not an error) if you vmotion between two hosts where the vmkernel vmotion port group is named differently, but this is probably more a “heads up” to the admin in case he is in fact trying to move a VM through a (relatively) slow layer 3 device by mistake.

Also the whole “devices per vlan” discussion is pretty dumb imho. The problem with a large broadcast domain is mostly linked to the actual broadcast traffic cumulated by having a large number of “chatty” devices share a network. Since the vmkernel does not try to autodiscover anything, it only needs to send broadcasts when initiating an IP connection to a device whose MAC-address is not already known. This results in one or two “arp who-has” packets, and then is shuts back up again.

Sticking with that thought, let’s assume that you configure a dedicated vlan for vmotion (which is a good practice in medium+ size installations anyhow). When do you get broadcasts? only whenever an ESX host is starting a vmotion and needs to map the target ESX’ vmotion IP to a MAC address. Even if that host fails to respond you only get one broadcast every second or so, until the vmotion times out. So what’s that? Maybe 30 broadcasts in as many seconds.

So I would think that you can stick a VERY large number of ESX hosts on the same subnet without incurring any significant overhead due to broadcast traffic. Probably a more relevant limit would be how large the mac-address-table on your switches can grow, but these days we are probably talking at least 4 digits.