I recently wrapped up some testing in my lab around VMware HA; specifically, around VMware HA isolation response. My tests involved various network configurations and attempted to clearly document the behavior of VMware HA isolation response under different circumstances. I thought I’d share some of my findings here in the hopes that others would find this information useful as well. (Keep in mind that some of the stuff listed below is just common sense, but I’m including it here anyway just for completeness.)
- Ensure that the vSwitch hosting the Service Console has at least two uplinks. Keep in mind that instead of leaving that second uplink primarily unused, you can place other traffic on the same vSwitch and use the “Override vSwitch failover order” option to direct traffic preferentially onto certain uplinks. (I’ll most likely post a separate blog entry about that so that I can explain that in more detail.)
- Ensure that DNS is working correctly on all ESX hosts in the HA-enabled cluster. You should verify host name resolution for both short names as well as fully-qualified domain names (FQDNs). Although I’ve seen numerous recommendations to hard-code entries into /etc/hosts, this approach is difficult to manage and does not scale well. Just fix DNS instead.
- Ensure that the Service Console’s default gateway responds to ping. If it does not, you’ll need to use the das.usedefaultgateway and das.isolationaddress parameters to change where VMware HA should check to see if it is isolated. Chad Sakac recently discussed these items as well, so check that entry for additional information.
- In a Cisco networking environment, ensure that Portfast is enabled on all physical switch ports. This will help reduce the possibility of an isolation response occurring due to transient network issues. Otherwise, the delay to put the port into a forwarding state is longer than the isolation response timeout, and a brief loss of connectivity could easily result in triggering VMware HA isolation response.
- If you are going to use a second Service Console port, be sure to specify a different IP subnet for the matching vswif interface. Otherwise, the Service Console’s routing table gets involved and tries to route everything through vswif0. That kind of defeats the purpose behind the secondary Service Console. My tests showed that isolation response was triggered every single time connectivity to vswif0 was lost when the secondary Service Console shared the same IP subnet as the primary Service Console interface.
- It should go without saying, but be sure that the secondary Service Console port is placed on a different vSwitch than the primary Service Console port. (Common sense, I know, but it’s worth pointing out anyway.)
- My tests have not shown that it’s not necessary to use a secondary isolation address when using multiple Service Console ports. The same post by Chad I linked to earlier seems to imply (unless I’m reading it incorrectly) that you should have multiple isolation addresses. I’m certainly open to any additional clarification any readers may be able to provide.
If you have any additional information or recommendations to share, please include them in the comments.
Tags: ESX, Networking, Virtualization, VMware, VMwareHA
-
Hi Scott,
I hope read this right, the whole subject can be a confusing thing.
Point 1 - The potential problem I see with this, is that I don’t think VMware recommends sharing SC uplinks with other traffic, for security best practice.
Point 5 - Very interesting. I would almost regard this as a bug. Obviously a second subnet is preferable for extra redundancy of certain non-VMware components, but the hosts should be able to try a second SC connection in the case of the first’s failure, even if it is on the same subnet.
Point 7 - double negative
If I read this right, I would agree that you shouldn’t need a second one IF the second SC is on a different subnet. However, if they are both on the same subnet, then you would want multiple addresses, because the default of both would point to the same thing (the DG). Hope I made sense with that one 
Keep them coming Scott, this is a very interesting area. Forbes.
-
Forbes,
You are definitely correct about one thing–VMware HA can be confusing at times!
Point 1 - You are correct, from a “security best practice” the Service Console should be segregated from other types of traffic. But we live in the real world, and many times in the real world we have to carefully balance risk vs. reward. In many implementations, we simply don’t have enough NICs to provide two dedicated NICs to the Service Console when one of those NICs is almost GUARANTEED to go almost completely unutilized. Since most users agree that the use of layer 2 VLANs is “secure enough,” using VLANs to segregate traffic on a vSwitch still allows us to host multiple port groups on the vSwitch, provide redundancy for multiple types of traffic with fewer NICs, and still provide a reasonable level of security.
Point 5 - I wouldn’t call it a bug, but I would say that VMware could do a better job of handling it. The SC routing table shows both interfaces as possible routes to the subnet, but vswif0 is listed first and therefore is tried first, even if the uplinks for vswif0 are down. Hence, you have to use a different IP subnet. If there were some way of notifying vswif0 that the vSwitch uplinks were down (sort of like the link-tracking functionality some blades chassis use), that would fix the problem, I think.
Point 7 - Well, having a second SC on the same subnet with a second isolation address would be useless because of the routing issue I described above. ESX would be unable to reach either isolation address (the default or the secondary) because it would try to route via vswif0, which is down. The only time a secondary isolation address would be useful is when the primary SC and secondary SC are on different IP subnets, and my tests show that–even in that configuration–a secondary isolation address is not required in order to prevent VMware HA isolation response.
Thanks for reading! I hope my responses made sense; feel free to ask more questions.
-
About point 5:
It’s not a VMware bug, it’s related to the RedHat Kernel they use. For more info check the following link: http://theether.net/kb/100041An additional isolation address could be useful when your gateway dies for whatever reason.
If you specify two service consoles you will be sure that you won’t have Spanning Tree problems. Because you will have two mac-adresses which will always stay attached to the same port. Besides that you can set the response time to 20 seconds instead of 30 with a redundant nic setup. This can save you 10 seconds in case of a complete failure. Which is a lot in my opinion.
HA is one of the main reasons I wrote the SC redundancy blog a while back:
http://www.yellow-bricks.com/2008/01/14/service-console-redundancy/
-
I do agree with your points Mike. However the one thing I am curious about, because I have seen several configurations like this, is why you would add a second uplink to your Primary SC connection? I’ll explain a bit further.
I’ll make a couple of assumptions to simplify things:
- The first SC is connected to vSwitch0 which has a single uplink.
- On the host you always plan to run VMs which connect to a vSwitch with a least one uplink, and this vSwitch is separate from your first SC vSwitch. Let’s call it vSwitch1.So here’s the point. You create a second SC connection on a port group which is connected to vSwitch1. You ensure that this port group is on a different subnet from the first SC – for the reasons you have already explained. So why would you need a second uplink on vSwitch0? You already have the physical redundancy because vSwitch1 has at least one uplink.
Forbes
Actually, typing this out and formalising my reply has made me realise one potential reason - if you are in an environment without VLANing.
-
One other thing to keep in mind, HA uses all service consoles, and there’s no way to tell it to ignore certain ones. So if you are using iSCSI and have your iSCSI network completely separated from the rest of your network, (since you need a service console on the iSCSI vSwitch for finding the LUNS and CHAP authentication), HA won’t kick in if it only loses connectivity to one of the service consoles.
- Damian
-
Thanks as always for the solid post, Scott.
In general (and this is philosophical - I haven’t seen anything SPECIFIC as an issue, I’m just being conservative) - I would tend not to use the service console networks for any other traffic, since it’s the core management model. In my basement, I’ll use VLANs and be ok, and would consider it rationally as an option for a production environment - but I wouldn’t use the SC interfaces themselves for other traffic. I haven’t really thought through WHY - it gives me the heebie-jeebies.
Re DNS - AAM has always been very, very sensitive to proper DNS - but I’m with ya - the way to fix DNS is never to hardcode on a host - that’s asking for operational nightmares. Just do DNS right. What’s fun in a lab, and quick hack may work - but what’s a good idea in production is often different.
My comment on the second isolation address in my blog entry was for the reason that Duncan stated - purely because in my experience at least, it’s more likely that there is a momentary problem with the gateway (or any single IP) than there is of a cascaded network problem affecting multiple interfaces on the ESX server. And since the (right) default behaviour is to shutdown the VMs - possibly affecting service in a big way, it certainly doesn’t hurt.
For exmaple - if you have seperate subnets (you are 100% correct on the vswif0 routing “issue” - and BTW, Duncan’s response is right on the “why” also) for service console, and the implied two isolation addresses (I wouldn’t ever want the isolation address to be routed, so of course it has to be on that subnet), you will ONLY get isolation response if there is REALLY a big problem, not a cabling problem (likely a catrophic switch failure, or a CatOS upgrade gone horrifically awry - both of which are ridiculously rare)
Just my 2 cents….
-
Chad,
I appreciate your input. I disagree with you with regard to the Service Console, though. I’ve seen way too many VI3 installations where we simply didn’t have enough NICs to dedicate two of them–on one vSwitch or on two separate vSwitches–to just the Service Console. My discussions with various security folks also indicate that most organizations consider layer 2 VLANs to be “secure enough” for management traffic.
With regard to the isolation address, it’s important to note that the isolation address only comes into play when the VMware HA agent loses heartbeats from the other ESX hosts. At least, that’s my understanding. This means that even if you don’t have a second isolation address on a second Service Console connection, as long as heartbeats are still be received HA will not trigger an isolation response. This plays into my statement about not *needing* additional isolation addresses in order to take advantage of multiple Service Console connections. I could be wrong, though…feel free to correct me if I am.
Thanks!
-
Scott - thanks for the impetus, it made for a good day of exploration (amongst the usual cast of daily chores). A summary of the learning is here:
http://virtualgeek.typepad.com/virtual_geek/2008/07/vm-ha—service.html
Second - I wasn’t explicit enough in my earlier comment.
“In my basement, I’ll use VLANs and be ok, and would consider **VLANs** rationally as an option for a production environment - but I wouldn’t use the SC **physical** interfaces themselves for other traffic. ”
VLANs are **perfectly** fine. Personally, I would go this way in blade cases only - otherwise, its clean, simple, and not particularly extra cost for a physical interface.
Now - the raison-d’etre for multiple isolation addresses (and why my view is that this should be a default) is twofold - even if you have two seperate subnets for your service consoles (as you’ve discovered), by definition one of them (or both) will have to route to get to the isolation address. Keeping the isolation address on the subnet means there’s no router in the mix (KISS being goodness). Having two also protects you against a single isolation address being unreachable (often caused by spanning tree issues).
-
Chad,
Good write-up, thanks for the work on this topic. VMware HA is definitely under-documented.
I can see your point about multiple isolation addresses; I guess my point is that I don’t see where multiple isolation addresses REALLY help. Understanding that HA isolation response only kicks in when host-to-host heartbeats are lost AND when the isolation address is unreachable, then providing a second Service Console interface on another subnet makes it VERY unlikely that all host-to-host heartbeats would be lost. Since the host-to-host heartbeats continue, the HA agent never relies upon the presence (or absence) of the isolation address.
Still, it certainly doesn’t hurt to include one…and if it can provide a small improvement in redundancy then I guess it’s worth it, eh?
-
Great article.
I still feel we’re just getting started with ESX. HA and isolation is an area we’ve not really addressed yet.
On a side note I have to admit we were advised to add ESX servers to /etc/hosts although DNS is also working correctly.
However in my hosts file I specify long and short names. Will DNS handle resolving of shortnames? Will the ESX servers apply their DNS suffix when attempting to resolve shortnames?
-
Chris,
It’s very common to hear of users having stuff hard-coded in /etc/hosts. You’re not alone in that regard.
DNS will resolve both short and long names. However, you must ensure that your ESX server is properly configured with the right domain name so that short names are placed in the right DNS domain for resolution. This can be done most easily in the VI Client.
-
Thanks scott. I’ll make sure it uses DNS now. It wasn’t too bad with 4 hosts, but now 12 and soon ??
-
Wow– thanks for the in-depth knowledge everyone. If I could take it down a notch I want to make sure I am not missing something architecturally based on what I am reading here.
I have 5 hosts, each with 2 nics for a total of 4 ports. am using a port on each nic for a total of 2 ports as a virtual switch to which I connect all my vm’s.
I have another virtual switch that contains my virtual console as well as a vmkernal port for my NFS connectivity. This vswitch is mapped to 1 port on one of the nic’s.
Finally, my last nic port is mapped to a 3rd virtual switch which is for vmotion. This virtual switch contains a service console (on a different subnet than the other service console) and a vmkernel port for vmotion.
With the exclusion of the last nic port which is for vmotion, the other 3 nic ports are connected to my switch as trunk ports where I am tagging about 5-6 different vlans (DMZ, iSCSI, etc…).
Everything seems to work fine, but when I read an in-depth posting like this I wonder I am possibly missing something.
Thanks everyone!!
-
Hook,
If I’m understanding your setup correctly, the big problem I see is that you have only a single uplink for your Service Console connection. If that connection goes down, you’ll trigger isolation response. Depending upon what version of ESX you’re running, isolation response may be set to power down all VMs.
If it were me, I’d combine the Service Console port with the VMotion traffic (unless you have a requirement of a physically segregated VMotion network) and use NIC failover ordering to prioritize the NICs between the two port groups. This provides failover while minimizing the number of NICs required.
If you do have a requirement for a physically segregated VMotion network, then there’s a setting in recent versions of ESX that allows VMware HA to leverage the VMkernel ports for heartbeats as well. That would be an option to explore. Check out Duncan Epping’s site (http://www.yellow-bricks.com) for more information on that setting.
Hope this helps!
-
Hi,
Two Servers with ESX 3.5 Update1, I have created a Virtual Switch with those two
1) VMKernel port for VMotion (reverse calble between servers)
2) Service Console 2Put everything in the DNS and everything seems to work OK. I known that I cannot access the console on that switch, but just in case someone unplug the service console 1 network cable, HA will not transfer all VMs to the second server
-
Hi,
I would like to see a similar post of how the upcoming VMware FT will react to these failures and isolations as well. Though all I can find about that is a small post and a nice video at http://www.virtualizationteam.com/virtualization-vmware/vmware-esx-40-ft-fault-tolerant-sneak-peek.html
A better post of how FT will act toward these type of failures will be a hit.Enjoy,
Virtualization Master -
Hi Scott,
great article, i want just to add that, related to HA, there are an issue if the name of the ESX are in UPPER CASE. Yes, i agree with you that for a linux user is normal to write in lower case but some times i have find installation with trouble for that reason as i have reported in my blog http://gmercatante.myblogtwo.com/thanks.
-
This has been mentioned a few times, I think, but thanks for bringing it up again. Making sure that everyone knows about these little “gotchas” is important. Thanks!
Site Sponsors
18 comments
Comments feed for this article
Trackback link: http://blog.scottlowe.org/2008/07/11/vmware-ha-configuration-notes/trackback/