blog.scottlowe.org

In his three recent articles about Quick Migration and live migration, Jeff Woolsey spends a lot of effort differentiating Quick Migration, VMotion, and VMware HA. Personally, I thought the distinction between these features and the purposes they were intended to serve were pretty clear already, but apparently Microsoft’s earlier claims that Quick Migration and live migration were comparable confused everyone.

The three articles from Jeff can be found here:

Hyper-V Quick Migration and VMware Live Migration, Part 1
Hyper-V Quick Migration and VMware Live Migration, Part 2
Hyper-V Quick Migration and VMware Live Migration, Part 3

In the first article, Jeff discusses the importance of high availability (HA) in virtualization scenarios. He’s absolutely right on target with his statements: HA is critical in virtualization implementations. I couldn’t agree more. VMware recognizes this fact and includes VMware HA, and Microsoft recognizes this fact and provides integration between Hyper-V and Windows Server 2008 Failover Clustering. So far, so good.

In part two, Jeff goes on to state that VMotion doesn’t work for unplanned downtime. Again, he’s absolutely correct: VMotion doesn’t work for unplanned downtime. Then again, apart from the comments that Jeff claims to have received from VMware supporters stating VMotion was “far superior for unplanned host downtime and that it was a much better HA solution”, I don’t think anyone has ever claimed that VMotion was an HA solution. I know I certainly haven’t. I can’t recall VMware ever making that statement. After all, if VMotion were an HA solution, why would VMware have VMware HA? What point would there be in two different HA solutions?

Further in that same article, Jeff compares VMware HA with Windows Server 2008 Failover Clustering, aka Quick Migration, and states that they are comparable technologies. Once again, I agree; VMware HA and Quick Migration are comparable technologies. Both will restart virtual machines on another host automatically in the event of a host failure. OK, Jeff and I still agree thus far.

Part three of Jeff’s series wraps things up by attempting to downplay the importance of VMotion. In his words, “Even customers with Live Migration still wait until off hours to service the hardware.” Unfortunately, this is where Jeff and I have to disagree. I don’t know how many customers they spoke to, but I know I have customers that have live migration functionality who use it during business hours. Besides, live migration isn’t just about hardware servicing or patching the root partition/parent partition/Service Console, it’s also about enabling dynamic load balancing a la VMware DRS, or enabling power savings in off-hours via DPM. After all, just because you can shut down and/or power off guests to migrate them doesn’t mean you will necessarily want to in every instance. It may be acceptable for some workloads, but not for all workloads.

I just can’t help but feeling that if Microsoft hadn’t made the comparisons between VMotion and Quick Migration themselves earlier in Hyper-V’s development, this sort of “unequal comparison” that Jeff is trying so hard to clear up wouldn’t have happened.

In The Four Horsemen of the Virtualization Security Apocalypse, Chris Hoff shines a great big spotlight on the dark side of virtualization security (or virtsec, as its increasingly being known). To quote from Hoff’s article:

Short of the notions I’ve discussed previously regarding instantiating the vSwitches into hardware and loading physical servers with accelerators and offloaders for security functions, there aren’t a lot of people talking about this impending set of challenges or the solutions in the short or long term.

This should be cause for alarm.

These issues are nasty. Combined with the organizational issues of who actually owns and manages “security” in the virtualized context, this stuff makes me want to curl up in a fetal position.

I agree with what Hoff has to say and I’m glad he’s taking the time to boil down the issues so that non-security-minded IT pros can really understand the problems. However, Hoff, I have to take you to task for one thing in your article: the kitten thing was just too much. Poor little kitten…

I particularly agree with Hoff’s #1 point (”Virtualized Security Screws the Capacity Planning Pooch”). The idea behind VMsafe and all these virtsec appliances is a great idea and all, but what about the overhead? At what point does having all this “extra” security so greatly bog down our virtualization engine that it’s no longer worth it to virtualize? And how do we actually, realistically begin to address this issue? Do we move the security functions into the hypervisor itself? And while this might address the performance concerns—although I don’t think so—isn’t this just instantiating Hoff’s vUTM?

One of the interesting things that I hope to be able to do soon is try to measure the overhead of some of the virtsec appliances that are currently available on the market. Not to publish any results or hit any vendors over the head with the information, but just to have a better idea for myself and my customers about how this stuff actually behaves in the real world. If anyone has already done that sort of thing and is willing to share their information with me, I’d be mighty appreciative.

I am curious about something—how many organizations are using a single physical host with VMs across different security zones? See, this is something that I would never recommend, and to me it seems like physically segregating your security zones into different virtualization environments solves a fair number of the concerns about the “dynamic data centers” created by VMotion, VMware DRS, and VMware HA. Or am I overlooking a critical aspect?

As a follow-up to my earlier post about using VLANs with VMotion, I wanted to also share a brief configuration snippet (based on a Cisco IOS-based switch) that aligns with the recommendations in that article. This is nothing new to those who are familiar with IOS, but for those readers who are not it may provide some helpful information.

For the purposes of this configuration, I’m making the following assumptions:

• VLANs 100 and 200 are the VMotion VLAN and some other production VLAN, respectively (perhaps a VLAN carrying virtual machine traffic, or a VLAN carrying Service Console traffic)
• VLAN 4094 is the “ESX Trunk” VLAN, which isn’t used anywhere else in the network for any purpose

Here’s the recommended port configuration for ports connecting to an ESX Server:

interface g0/1
switchport trunk encapsulation dot1q
switchport trunk native vlan 4094
switchport mode trunk
switchport trunk allowed vlan 100,200,4094
switchport noneg
spanning-tree portfast-trunk

Technically, the “switchport noneg” command won’t really do anything; it disables DTP (Dynamic Trunking Protocol) but DTP isn’t supported by ESX Server anyway.

A couple of notes about this configuration:

• Refer back to my article on the native VLAN with ESX Server; by setting the native VLAN to 4094 (the “ESX Trunk” VLAN), you won’t carry that traffic into the ESX Server. If used on a vSwitch that also carried Service Console traffic, then it could impact automated build scripts.
• If you needed to combine this configuration with EtherChannel, refer back to my original article on ESX Server, NIC teaming, and VLAN trunking.

Keep in mind the other recommendations as well: explicitly control trunking to other devices and explicitly control the transmission of VLANs across trunks to control “VLAN leakage”.

CCIEs and other experts, I’d welcome any other suggestions as well as recommended commands to use in the switch configuration to help maximize security and minimize exposure to VLAN-based attacks.

Xensploit, as it’s called, is the recently demonstrated exploit that allows virtual machines (VMs) that are “in flight” during a live migration (XenMotion in Citrix XenServer, VMotion in VMware ESX Server) to be manipulated. If you haven’t yet read the PDF that describes Xensploit, I highly encourage that you take a look at it. It’s very enlightening as to exactly what can be done to an in-flight VM.

Naturally, the best way to protect against this particular problem is to guard the integrity of the live migration network. For simplicity’s sake, I’ll refer to this as the VMotion network from this point on, but keep in mind that it is equally applicable to any network connections on any virtualization platform that uses live migration.

The most surefire way to protect the VMotion network is to place it on its own dedicated, physically separate network, using separate physical NICs plugged into separate physical switches that do not possess any connections to production networks. This will ensure that unauthorized access to the VMotion network is prevented, but comes with disadvantages as well: this configuration requires more physical NICs and more physical switches than other configurations.

In implementations with limited numbers of physical NICs, however, this isn’t really an option. In these cases, the use of Layer 2 VLANs and multiple port groups on a single ESX Server vSwitch to allow VMotion traffic to share the same physical NICs and the same physical switches as other traffic is a very common solution. In fact, it’s a solution that I’ve recommended many, many times. But does this configuration provide enough protection for the VMotion network?

The real question is, does a simple Layer 2 VLAN offer enough protection? That question, in turn, spawns other questions: what kinds of attacks are there against Layer 2 VLANs? Is it possible for traffic to hop across VLAN boundaries?

Armed with those questions, I set out to do some research. You can see some links in my del.icio.us bookmarks that pertain to the research I did. Basically, I found that Layer 2 VLAN attacks boil down to two basic types:

1. The first type involves a malicious host pretending to be a switch and forming a 802.1Q VLAN trunk with the real switch, which then passes traffic from all VLANs across to the malicious host.
2. The second type involves double-tagged 802.1Q frames and the native VLAN, whereby traffic can, under specific circumstances, hop from one VLAN to another without any Layer 3 routing involved.

(Network and security gurus out there feel free to elaborate or correct me on this information.)

The best way to address attack vector #1 is to explicitly disable automatic trunk negotiation on all ports that don’t need to be trunks. From my research and my (relatively) limited knowledge of Cisco IOS, this command should do it:

switchport mode access

This explicitly forces the switchport into a state where it will not negotiate an 802.1Q VLAN trunk with another device, hence killing attack vector #1 dead in its tracks. Again, this should only be done on the ports that are not connected to the ESX Servers; otherwise, you’re shooting yourself in the foot. Keep in mind that uplinks to other switches, ports going out to IP phones, etc., may also need to be configured as VLAN trunks. Really, the issue is about controlling the creation of unauthorized VLAN trunks in order to control VLAN leakage. One of the CCIEs at the office mentioned a “switchport noneg” command, but I’m not familiar with that one; anyone have more details?

Addressing attack vector #2 is also relatively straightforward. Since the VLAN hopping exploit takes advantage of the nature of the native VLAN (which I’ve discussed before here), setting the native VLAN on the trunk ports connecting to the ESX Servers to a VLAN that is not used anywhere else in the network will prevent VLAN hopping. For example:

switchport trunk native vlan 4094

From my NIC teaming and VLAN trunking article (one of the most popular articles on the site, by the way), you’ll see that I recommended at that time the creation of a VLAN that is used only as the native VLAN for 802.1Q trunks. At that time, I didn’t fully understand why; now, after additional research, I understand why I needed to do that and I also recognize that the suggested configuration provides a layer of protection against VLAN hopping attacks.

To summarize:

• To protect against malicious hosts forming unauthorized 802.1Q trunks, disable automatic trunk negotiation and explicitly/manually create trunks. The key here is to ensure that VLANs don’t inadvertently “leak” beyond where they should (also see note below about specifying the allowed VLANs).
• To protect against VLAN hopping, create a VLAN that is used only as the native VLAN on the 802.1Q trunks connecting to the ESX Servers. This VLAN must not be used anywhere else in the LAN. Set this VLAN as the native VLAN on the trunks into the ESX Servers.

In addition, my networking mentors also recommended the “switchport trunk allowed vlan” command to specify which VLANs are allowed to cross 802.1Q VLAN trunks. This will help ensure that VMotion traffic is limited to only those switches that absolutely must carry it; again, we’re seeking to control VLAN leakage.

With these configurations in place, using a Layer 2 VLAN to carry VMotion traffic on the same physical NICs and same physical switches as other types of traffic is fairly well-protected against malicious interference. While it is not as secure as a physical separate, dedicated network, it is secure enough for most organizations and the reduction in infrastructure needs generally outweighs the risks.

More information and discussion about Xensploit—and protecting against Xensplot—at the following links:

Keeping Your VMotion Traffic Secure
Two vulnerabilities found in VMware virtualization products
‘Live’ VMs at Risk While in Transit
News Flash: If You Don’t Follow Suggested Security Hardening Guidelines, Bad Things Can Happen…

UPDATE: I’ve updated the wording above to more properly reflect the goal behind the use of the “switchport mode access” command, and when it should be used. Colin, thanks for the feedback and clarification!

An issue has been discovered that can cause a CPU spike when VMotion is used in a DRS-enabled cluster with ESX Server 3.5 and VirtualCenter 2.5.  It is my understanding that this can occur with both DRS-initiated VMotion operations as well as manually-initiated VMotion operations.

Fortunately, there’s a workaround for this issue which involves editing the vpxd.cfg on the VirtualCenter server.  Full details on the change that needs to be made, as well as on the log entries that you should see on ESX Server afterward, are found in this article.

UPDATE:  It appears that VMwarewolf has had to pull the original article that described this problem and the workaround.  Fortunately, Google has a long memory, and here’s the workaround for this problem.

Immediately after the “<vpxd>” line in vpxd.cfg, add the following lines:

<cluster>
<VMOverheadGrowthLimit>5</VMOverheadGrowthLimit>
</cluster>

I’m guessing that this information may not be information that VMware wants easily disseminated to the world, or that the workaround has not been fully and completely tested.  So, use this information at your own risk.

In the meantime, this Google search will return the now-unavailable page; use the Cached link to see the workaround and the details from the log file that will help troubleshoot the problem.

UPDATE 2:  VMware has now published this KB article about the issue, along with the workaround for the problem.

I was supposed to be in this session!  I’m so mad…I had another obligation, someone to whom I’d committed to do something, so I couldn’t make the 5:00PM session yesterday during which VMware announced “Enhanced VMotion,” which is designed to alleviate some of the CPU compatibility concerns that have become a bit more visible in the last few months.

In reading the SearchServerVirtualization.com article on this announcement (which is the only reference I’ve seen thus far to the new capabilites), it looks like this may build on top of already-announced enhancements by Intel and AMD (Intel calls theirs FlexMigration, and I think AMD is calling theirs AMD-V Extended or similar).  No specific information was provided on a timeline, but given that ESX 3.5 is currently in beta it sounds like a likely addition to that product’s feature set.

I’ve discussed VMotion compatibility issues here before, so I’m glad to see both hardware vendors as well as VMware addressing these potential concerns.  As I have the opportunity to gather more information, I’ll update this post.  In the meantime, anyone with additional information is asked to add their info in the comments.

In an earlier article about VMotion compatibility, I mentioned that SSE support was a VMotion boundary, i.e., a host with SSE3 CPUs and a host with SSE2 CPUs would not be able to VMotion guests between them without first masking the SSE support bit.  I used this technique myself in our lab (more info in this article), and this issue drew some attention as the result of a presentation at VMworld last year.

Now, according to this posting on the VMTN forums, the introduction of the Intel quad-core CPUs has introduced another VMotion boundary, and that is the introduction of SSE4.  What this means is that you won’t be able to seamlessly integrate both dual-core and quad-core CPUs into a DRS cluster, because VMotion won’t work between these CPUs.  The only way to move systems from a dual-core SSE3 CPU to a quad-core SSE4 CPU will be a cold migration.  Having now been spoiled by live migrations with VMotion, I suspect a lot of administrators will be more than a bit perturbed by this.

It’s odd that this comes up—I was just speaking to some customers a couple of days ago about VMotion boundaries and CPU compatibility, and I commented that I was not aware of any major developments in CPU capabilities that might introduce new VMotion boundaries.  I guess I was wrong!

I have a feeling that VMware is going to need to resolve this kind of issue sooner rather than later.  As the hardware vendors race with each other to add new features and new virtualization support, we are going to see more and more artificial VMotion boundaries being introduced, and this will eliminate much of the flexibility that VMware currently offers organizations since they will have to resort to buying exactly identical systems.  The sooner that VMware can move into a position of being able to support custom CPU masks, the better off they’ll be in my opinion.

Richard Garsthagen of www.run-virtual.com has published an application (I believe he’s calling it VMotion Info) that clearly identifies the differences between CPUs across servers in a farm of ESX servers.  This application talks to the VirtualCenter server and then gather information from all the ESX servers represented in VirtualCenter.  After gathering all the information, it will then graphically present the differences and actually decode the raw bits to tell you exactly what the differences are (i.e., NX/XD supported/not supported, SSE2/SSE3, etc.).

This is extremely useful information.  The missing piece, unfortunately, is the conversion to the binary bits that are necessary to actually mask that information from the virtual machine.  I provided some of that information in my earlier article, but for completeness’ sake thought I might present it here again.  Hopefully, this will prove helpful to someone.

For example, here are the raw binary values I obtained on two servers in my server farm:

ID1ECX 0x0000641d (server one)
ID1ECX 0×00004400 (server two)

Converted to binary, that becomes:

0000 0000 0000 0000 0110 0100 0001 1101 (server one)
0000 0000 0000 0000 0100 0100 0000 0000 (server two)

Observing the binary values, we can easily see the differences between the two CPU families and why (by default) VMotion won’t work across them.  But what do these values mean?  And how does this information play into Richard’s VMotion Info tool?  Good questions.

Taken from this Intel document, here are the meanings of those binary values (bit 0 would be the rightmost bit in the binary values above):

0 - SSE3
1 and 2 - Reserved
3 - MONITOR
4 - DS-CPL
5 - VMX (Intel Virtualization Technology)
6 - Reserved
7 - EST (Enhanced Intel SpeedStep Technology)
8 - TM2
9 - SSSE3 (Supplemental SSE3)
10 - CID
11 and 12 - Reserved
13 - CX16
14 - xTPR
15 through 17 - Reserved
18 - DCA (Direct Cache Access)
19 through 31 - Reserved

In the example provided above, the differences between the CPUs were in bits 0, 2, 3, 4, 10, and 13. That translates into differences in SSE3, MONITOR, DS-CPL, CID, and CX16 support between the two CPUs.

However, not all those features are necessarily significant to VMotion.  In order to know what features are significant to VMotion, we’ll need to review the default CPU mask that’s provided by VirtualCenter to new virtual machines.  Based on my research, the default mask provided to a new Windows Server 2003-based virtual machine is this:

RRRR RRRR RRRR RRR0 00XR R0H0 000H 0RRH

According to the legend from VirtualCenter, any bit marked R or H must match for successful VMotion.  This means that although bits 0, 2, 3, 4, 10, and 13 are different, only bits 0, 2, 4, and 10 are actually significant to VMotion (bits 3 and 13 are already marked 0 [Hide feature from guest software] or X [Unused by guest software]).  In order to get VMotion working, we’ll need to mask those bits that are different and are significant to VMotion compatibility.

In addition, there are multiple registers—ECX and EDX—and different values (0, 1, 80000000h, and 80000001h).

I’m assuming that Richard’s tool only identifies those flags that actually impair VMotion compatibility.  Those flags, and their corresponding register bits are listed below:

NX/XD - Level 80000001 EDX bit 20
FFXSR - (unknown)
RDTSCP - (unknown)
SSE - Level 1 EDX bit 25
SSE2 - Level 1 EDX bit 26
SSE3 - Level 1 ECX bit 0
SSSE3 - Level 1 ECX bit 9
CMPXCHG16B - Level 1 ECX bit 13

I couldn’t find the feature flag values for FFXSR and RDTSCP in the Intel documentation.  Not shown above (as far as I can tell) is the flag for the Intel 64-bit extensions, which is found at Level 80000001 EDX bit 29.

So, if after running the Virtual Info tool you find differences in NX/XD and Intel 64 support, then you can create a custom CPU mask that hides (using 0 in the mask) level 80000001 EDX bit 20 and level 80000001 EDX bit 29.  By placing a zero in the mask, then these values are hidden from the VM and therefore won’t affect VMotion compatibility.

Using the Virtual Info tool, you can now easily and clearly identify the differences in your CPUs.  Then, using the breakdown listed above, you can easily create the necessary CPU masks to hide those differences and increase VMotion compatibility across hosts.  Keep in mind, though, that most of the CPU masks are currently unsupported by VMware.  (I just noticed that the supported vs. unsupported “relaxations,” or masks, are listed at the bottom of Richard’s tool.)

Also, there’s an ongoing VMTN forum discussion occurring about CPU compatibility and VMotion; Mike Laverick (of RTFM Education) also mentioned it on his weblog as well (see the section marked “New Community Initiatives”).

First and foremost allow me to make this disclaimer:  Do not do this on production systems.  While it may work for testing (I haven’t seen any ill effects yet), it’s definitely not recommended for production use.  I seriously doubt that VMware would provide any support for this kind of hack.

Now, with that serious warning out of the way, let’s get into the details.  Based on the information found in this VMTN forum discussion, I was able to find the information I needed to determine exactly what differences were causing VirtualCenter (VC) to report that it couldn’t VMotion between the two hosts.

The real trick here is the use of the CPUID ISO image, found on the ESX 3.0 CD.  Burn this ISO image to a CD and then boot from the CD; the results from this command will give you detailed information on the exact functionality and capabilities of the CPUs in your system.  Write this information down exactly as it appears on the screen when running the test; you’ll need it later.

For example, on one of the hosts I received information like this:

ID1ECX 0x0000641d
ID1EDX 0xbfebfbff
ID81ECX 0000000000
ID81EDX 0x20000000

This all looks like jibberish, but using this document from Intel gives the breakdown on what those numbers mean.  On a second ESX host, I received these values from CPUID:

ID1ECX 0x00004400
ID1EDX 0xbfebfbff
ID81ECX 0000000000
ID81EDX 0000000000

As you can see, the differences lie in the ECX register, which is exactly what VC was reporting during its “validation” process.  Having now identified the specific differences between the hosts, we proceed to mask (or hide) those values from the guest VMs by modifying the VM’s configuration.

For the VM that you want modified, right-click and go to Edit Settings > Options tab > Advanced > Advanced button.  From here, you’ll see a listing of the CPU registers (EAX, EBX, ECX, and EDX) and the masks that will be applied to each register at various levels.  Because all that was returned by CPUID was ECX and EDX and levels 1 and 81, that’s all we need to worry about.

ESX Server will supply a default mask that shows or hides certain flags to the VMs.  We can modify that mask to hide additional values, thus increasing VMotion compatibility at the expense of possibly lower performance (because VMs won’t be able to take full advantage of some higher-level functions of the CPUs).  For example, the default mask for ID1ECX for a Windows-based host is this (there’s a “Legend” button in the same dialog box to explain what these values mean):

RRRR RRRR RRRR RRR0 00XR R0H0 000H 0RRH

To make this work, what we need to do is change the R or H to a zero (0) where the differences between the hosts are.  However, in order to really understand where the differences are between the hosts, you must first convert the hexadecimal values above to binary.  In my particular example, here are the converted values (again, for ID1ECX):

RRRR RRRR RRRR RRR0 00XR R0H0 000H 0RRH (mask)
0000 0000 0000 0000 0110 0100 0001 1101 (first system)
0000 0000 0000 0000 0100 0000 0000 0000 (second system)

The only bits we need to mask are the bits not already masked (i.e., those marked with an R or an H in the mask) and where the values are different between the two systems.  In this case, bits 0, 2, and 4 are different and are not already masked.  (You’ll note that bit 14 is also different between the two hosts, but it’s already hidden in the mask and therefore is not an issue.)  To mask these bits, we’ll edit the line in this dialog box to put a “0” in the position for bits 0, 2, and 4, leaving all other bits intact (use a dash for all the other bits).  When you’re done, the custom mask in the dialog box will look something like this:

---- ---- ---- ---- ---- ---- ---0 -0-0

When this custom mask is added to the default mask, you get an effective mask that hides the different bits between the hosts:

RRRR RRRR RRRR RRR0 00XR R0H0 000H 0RRH (default mask)
---- ---- ---- ---- ---- ---- ---0 -0-0 (custom mask)
RRRR RRRR RRRR RRRR 00XR R0H0 0000 00R0 (effective mask)
0000 0000 0000 0000 0110 0100 0001 1101 (first system)
0000 0000 0000 0000 0100 0000 0000 0000 (second system)

As you can see, because none of the different bits between the two hosts are marked with an R or an H, then they are all masked from VirtualCenter and therefore won’t prevent VMotion from working.  (Keep in mind that an X means it’s unused or ignored; only the R and the H will come into play for computing VMotion compatibility.)

So far, I’ve tested this on VMs running Windows Server 2003, Windows 2000 Server, Solaris 10, and Linux (CentOS 4.3).  I’ve not seen any problems thus far, and I’ve VMotion’ed these VMs back and forth a number of times trying to uncover some compatibility problems.  If I discover any, I’ll be sure to post more information here.

UPDATE:  Also see this article for more information and a link to a tool that has been released to help with this process.

UPDATE 2:  Thanks to an astute commenter, I’ve updated the article accordingly to note that bit 1 (the bits are numbered starting with 0) does not need to be masked, since it is the same between the two CPU families.