have 2 geographically different data centers
dedicated line between
2 esx servers at each location
each side will have its own san that replicates (every hour)

just stuck on deciding if both sides should be on same subnet
or use different subnets with scripting ip changes/dns updates

having the same subnet on both ends really seems like the simplest approach

but after reading these comments, having second thoughts

OK, thanks for the clarification. You appear to be speaking from first-hand experience, so help me understand: where is the tipping point here? At what point does the number of VLANs tip the scales too far? Or if it’s not the number of VLANs but instead the breadth of distribution of those VLANs, what is the breaking point? And how much of this is mitigated by proper network design (which, I’ll grant you, not every organization has the pleasure to do adequately)? I want to understand your position more fully, so any information you can share would be great. You’re welcome to e-mail me separately if you’d like, and I’ll post the results of our separate discussion in a future blog post.

I am not talking about the actual requirement of L2 adjacency for the VM kernel ports. I agree with your position/comments — that particular requirement is of little practical concern.

The point I was trying to get across (as was Benik in his article — I think) is that in order for a VM “Guest” to be *usable* when moved from one VM Host to another (via DRS, manually, whatever) — there is a presumption that the source ESX host and the destination ESX host will both be able to trunk the same VLANs from their local network switch.

Example: If a web server (a VM guest) had a data NIC address of 192.168.1.1/24 in its current home (VM host) on one side of your datacenter — when it gets moved to a different VM host on the other end of your building — it still needs to retain that exact 192.168.1.1/24 address — thus both VM Hosts need access to the VLAN which holds that IP subnet.

Of course this type of move could be dealt with in other ways, such as scripting server IP address changes and updating DNS, etc. as part of the DRS or manual VMotion move process — but that’s just not the ‘seamless’ mobility everyone’s been promised by the virtualization sales vendors.

If you have a very large datacenter environment and VM is *everywhere* throughout it — there is really no practical way to have L2 ‘mobility’ throughout the entire facility enabling any VM guest to be moved to any VM host within any locale of the facility without running a ‘flat-earth’ (all L2 VLANs available anywhere) model that will undoubtedly get you into some serious trouble that increases the larger you become.

Planning is certainly key here. If you know in advance that you want one L2 environment to support DRS between servers located in three different rows, etc. of your facility, then you design accordingly up front. If you’re able to do this and live with those confines, then great. If your working in shops small enough not to have to concern yourself with such things, then that’s wonderful too.

Unfortunately, many large datacenters are not really all that well planned. VM sneaks its way in to the environment in random, unplanned ‘pockets’. Those pockets all suddenly ‘need’ to be part of the same farms/clusters Again, the expectation is out there (is often actively set by VMWare sales folks) that VMotion and DRS allows you to move any guest to any VM host in your facility like magic. Never mind the contortions the network is bent into on the back side to make that vision a reality. Never mind how unstable such an environment can silently become.

Take the VM part away from this discussion for a second, and think of this in terms of physical servers. If it was cost-effective to physically move servers around the datacenter (which it of course isn’t) — can you think of many large datacenters where you could just move a server from one row to another, one building to another, one campus to another, etc. and get to just take your (L3) addresses with you without any special arrangements in advance?

-Jeremy

It is not the VMotion process itself which imposes this requirement, but rather the very assumption that a virtualized server can be moved from a physical host at one end of the data center to a different host in a completely different geography within the same (or different) facility and retain its network addressing as part of the process. While this can certainly be achieved in smaller data center environments, as the size of the facility(s) increases — the requirement of having many common VLANs which ‘span’ the entire site infrastructure to permit L2 mobility on-demand is generally not considered to be a scalable practice that promotes network stability.

Most large enterprise data center environments must impose various L3 (routed) boundaries to break up L2 (switched) network environments into self-contained ‘blocks’ to meet scalability and fault isolation requirements demanded by most large enterprise organizations. L3 isolation within facilities may also be used to meet various security and regulatory requirements. These requirements are even more prevalent in service provider (‘cloud, etc’) environments.

While VMotion itself can occur through L3 boundaries, the stark reality is that the the IP addressing the VM had in its former location on one side of a L3 boundary is NOT going to be valid once that host moves to its new location on the other ‘side’ of that boundary.

The host can be re-addressed as part of the VM move, and DNS, etc can be updated to reflect its new location — but its far from the near ‘seamless’ move that can be achieved if the source and target were truly L2 adjacent. Network (and storage) architectures can certainly be tailored to allow L2 adjacency across large facilities in support of properly defined VM mobility requirements — however the common virtualization ‘sales pitch’ of any VM moving between any host in your facility (or remote facilities) any time you want — no problem” — is indeed quite a stretch. If you are in a *large enterprise* data center that allows complete L2 mobility between any hosts within the facility (i.e. every VM host anywhere in your facility can access any .1q VLAN via its data trunk interface(s) — then your network engineering team needs to start preparing their resumes, as they’re due for a major meltdown after provisioning such a large ‘flat-earth’ L2 model.

Before anyone jumps on that last statement, yes, newer network switch virtualization technologies like Cisco’s VSS/VPC/OTV can certainly help you build a monstrous flat L2 network with significantly less risk. However, everything has its breaking point, and when you mitigate STP concerns with cross-chassis etherchannel, etc. then perhaps your next barrier to scalability will be with MAC learining rates, or MEC table limitations, etc.

Violate the old mantra about building scalable and fault tolerant networks “route when you can, switch when you must” at your own (eventual) peril.

I’ve listed down simple questions and described a pretty common deployment scenario in the Java/JEE world. I’d greatly appreciate answers (no flaming pls, we are noobs):

1) The app running on the app server uses 80-90% CPU for 8 hrs a day
2) The app is really a cluster of let’s say 12 processes running on 4 machines. App server already does FT/ HA and all that stuff. That’s why it’s a cluster
3) Each process has been allocated and is using 3-4G of RAM
4) Yes, these are JVMs
5) 1G switch

Given these basic stats,
1) Would you put this setup on VMs? (Not to be confused with JVMs)
2) Obviously this vmotion has to use CPU. Someone has to be syncing the deltas
3) Again, it is also using the network and bandwidth
4) Are you saying there is no performance impact on the application? No context switching, cache pollution, interruption, bus contention

Or, is there a little elf sitting inside who can magically make this happen without impacting performance and guarantee 0 downtime?

Explicitly clustered apps like App servers, Grids and NoSQL software and other such apps that people have been using for years, provide many knobs to suit the user’s combination of performance/FT/HA etc. They do not hide this – http://en.wikipedia.org/wiki/Fallacies_of_Distributed_Computing nor this – http://en.wikipedia.org/wiki/Leaky_abstraction from users/developers.

Am I right or am I right? Perhaps someone can clarify such things for us? (without just asking us to RTFM pls)

Thanks!