Hardware

Some time last week, one of my NetNewsWire subscriptions turned up an article titled “HP VirtualConnect” from Brent Ozar. It turns out that Brent’s blog, while heavily SQL Server-oriented, has a few nuggets of HP Blade information:

HP C-Class Blade Chassis Review
HP C-Class Blade Chassis Review Part 2: The Cisco/Brocade Interconnect Switches
HP Virtual Connect

Of course, Brent must be pretty smart since he linked back to my site…

Seriously, though, have a look at Brent’s HP c-Class articles for a good overview of the product and its associated technologies.

This is a handy little trick that many of you may already know. Starting with version 3.5, VMware added support for Cisco Discovery Protocol (CDP) on the ESX Server vSwitches. CDP support is enabled on a vSwitch with this command:

esxcfg-vswitch -B both vSwitch0

The “-B” parameter is case-sensitive; the “-b” (note the lowercase B) displays CDP status while the “-B” (uppercase B) configures CDP.

Once CDP support is enabled on the vSwitch—and assuming it is enabled on the physical switch—running the “show cdp neighbor” IOS command will show the link between each physical switch port and the matching ESX Server NIC. The output will look something like this:

Capability Codes: R-Router, T-Trans Bridge, B-Source Route Bridge
                  S-Switch, H-Host, I-IGMP, r-Repeater, P-Phone

Device ID Local Intrfce Holdtme Capability Platform      Port ID
s3        Gig 0/26        147      T S     WS-C3524-XFas 0/24
esx04     Gig 0/22        168       S      VMware        ESXvmnic0
esx04     Gig 0/21        168       S      VMware        ESXvmnic1

As you can see in the output above, the CDP output clearly links the physical switch port and the ESX Server NIC. This makes it incredibly easy to identify the NICs in the server. This is particularly helpful in blade situations, since you can’t exactly unplug the NIC and see which one goes down with “esxcfg-nics -l” (a common approach to identifying the NICs in the server). Of course, this requires Cisco switches in the blade chassis. Since the internal port mappings on the blade chassis determine which NICs connect to which ports, this command adds the mapping within ESX Server and lets us quickly and definitively identify the NICs in the server as seen by ESX Server.

A friend and colleague of mine recently launched a blog of his own, The Blade Blog.  If the useful information that’s been posted there so far is any indication, this will be a definite addition to your RSS subscriptions.  Keep up the good work, Aaron!

UPDATE:  After some great feedback on the launch of his site, Aaron’s moved away from Blogger to a hosted WordPress solution and has gotten a new domain name.  You can now find Aaron at the Blade Vault.  More information and updated URLs available here and here.

For some good information on using HP VirtualConnect with VMware ESX Server, go check out this article just published by SearchVMware.com:

While the idea of server profiles can be useful in VMware Infrastructure 3 (VI3) environments, Virtual Connect’s ability to integrate with VLAN tagging configurations is perhaps more applicable to ESX Server deployments on c-Class blades. In this article, we’ll take a look at how VirtualConnect integrates with ESX Server with regards to networking configurations.

Of course, I’m a bit partial to this particular author (wink, wink), but it is a good article nevertheless.

Feel free to hit me up with any corrections, comments, or thoughts.  And be sure to go read my article, so TechTarget will keep asking me to write for them!

Here are some VMworld links that I’ve been collecting over the last day or so.  As I have time, I’ll try to expand upon them and add my own thoughts and views, but wanted to mention them here briefly, at least:

• VMworld 2007: Embedded ESX a can of worms:  Alex Barrett of SearchServerVirtualization.com comments on the release of ESX Server 3i and what impact it may have on the usage of ESX Server in the enterprise.  Certainly it will have an impact, but I think the impact will be greater for third-party vendors than for IT managers.
• VMworld 2007: ESX price is not the problem
• VMworld 2007: VMware users say, “Lower prices”:  So are prices an issue, or not?  The guys and gals at TechTarget seem to be reporting conflicting views.  I’ll have to delve into this in a bit more detail.
• Open Virtual Machine Format Specification (OVF) Submitted to DMTF:  The OVF was announced earlier at VMworld 2007, but no details have really surfaced (at least, none that I’ve seen yet—feel free to forward information you may have found on OVF).
• VMworld 2007: bigger, broader and Viridian:  VMworld, strangely enough, isn’t just about VMware, so here’s some information from Microsoft on their developing hypervisor, code-named “Viridian”.  Here’s another blurb about virtualization security as well.

I’ll try to add some more information and my thoughts on some of these issues as soon as possible.  In the meantime, I’m off to my final session here at VMworld 2007 on Day 2!

VMware has made a number of announcements today, jointly making the announcements in general sessions at Partner Day/Technology Day here in San Francisco, CA, this morning along with official press releases:

• VMware Unveils Next Generation Hypervisor to be Integrated in Server Hardware:  News of this was broken on the Internet (mostly via virtualization.info) several months ago, but now VMware has publicly announced the movement of their hypervisor into the server firmware.  This movement is expected to bring about a number of significant improvements, many of which were alluded to by Diane Green during her speech this morning and seconded by Scott Davis (Chief Data Center Architect at VMware and Virtual Iron co-founder) during his speech as well.
• VMware Introduces Next-Generation Disaster Recovery with VMware Site Recovery Manager:  VMware Site Recovery Manager attempts to bring about some automation and workflow to disaster recovery (DR) solutions built on top of VMware.  Personally, I haven’t seen a great deal of information on this product, although I knew about it under NDA before today’s public announcement.  I’m looking forward to seeing more details on this solution, as well as getting the opportunity to have a look at the technical portion of the product.  (You know me—I love the technical details.)
• VMware Enables Enterprise Desktop of Tomorrow:  Virtual Desktop Manager 2 is VMware’s first (public) foray into the VDI broker space, whereas they have previously relied upon Propero (whom they acquired), Dunes (whom they are rumored to have acquired), Leostream, Provision Networks, Citrix, and others.  Personally, I hope that VDM 2 is much better than the previous-generation Propero solution, which I found unwieldly, complex, and unreliable.  (Of course, your mileage may vary.)  From what I understand, it is expected that VMworld attendees can apply for the VDM beta at the conference this year.  I don’t know that for certain, but I’ll be finding out tomorrow when VMworld officially starts.

I also had the opportunity this morning to attend a great in-depth session on VMware Consolidated Backup.  I’ll try to post more information on that session later today.

All in all, it’s a busy time for VMware, who is taking this opportunity to solidify their lead in the virtualization market against up-and-coming competitor Xen (now owned by Citrix) and Microsoft.  As developments occur and as I get additional information, I’ll post more here.  Thanks for reading, and feel free to clarify or correct any of my information in the comments.

I just added another server to my VMware ESX Server farm, and since this farm is being built from leftover, donated servers that aren’t being used elsewhere (such is the curse for a non-revenue generating test environment), I don’t have the luxury of ensuring that all the servers in the farm have the same (or compatible) CPU families.  As I’ve discussed in earlier blog postings (Sneaking Around VMotion Limitations and VMotion Compatibility), we can use custom CPU masks to help address the issue of different CPUs in the same ESX server farm.

Because this is a non-production environment, I don’t have to worry too much about the fact that VMware doesn’t support these types of custom CPU masks.  With that in mind, I thought it might be helpful to again walk through the process of identifying the differences in the CPUs and creating custom CPU masks to address those differences.  Of course, I don’t recommend doing this in production environments.

The first step is to gather the information from the CPUs themselves.  Richard Garsthagen’s VMotionInfo utility should provide all the information you need, but I have found it easier to just get the raw data myself using the CPU ID boot CD.  This does require that you reboot the ESX server (thus taking it down), but given that we can use VMotion to move guests to a (known compatible) peer, this is not a huge issue in my mind.

Using the “Verbose” boot option of the CPU ID boot CD, here is the data gathered for the three servers in my farm.  The value in parentheses is the value returned by CPU ID; I’ve added the binary value after that for comparison.

Server1 ID1EAX (0x00000F43)  0000 0000 0000 0000 0000 1111 0100 0011
Server2 ID1EAX (0x00000F25)  0000 0000 0000 0000 0000 1111 0010 0101
Server3 ID1EAX (0x000006F6)  0000 0000 0000 0000 0000 0110 1111 0110

As you can see, converting the hexadecimal values (the ones in parentheses returned by the CPU ID boot CD) to binary shows that the differences lie in the last 3 bytes.  That in and of itself doesn’t really help that much until we combine the information above with the standard CPU mask for that register.  Here’s the same information again, only this time with another line showing the standard CPU mask:

Server1 ID1EAX (0x00000F43)  0000 0000 0000 0000 0000 1111 0100 0011
Server2 ID1EAX (0x00000F25)  0000 0000 0000 0000 0000 1111 0010 0101
Server3 ID1EAX (0x000006F6)  0000 0000 0000 0000 0000 0110 1111 0110
Standard CPU Mask ID1EAX     XXXX HHHH HHHH XXXX XXXX HHHH XXXX XXXX

The CPU mask shows that only the bits from byte 3 (third from the right) are significant.  This is indicated by the “H” in the mask, which (according to the legend in the Virtual Infrastructure Client) means that the guest will see the value of that register and that the value of the register must match for a successful VMotion.

<aside>I have not yet determined if the standard CPU masks are guest OS dependent, but I suspect that they are.  Be sure to check the standard CPU mask against a VM configured for the guest OS that you will actually be running, or it may not work as you expect.</aside>

To fix this, we need to add a custom CPU mask that masks that bits that are different.  Here’s the same information again, this time with a custom CPU mask and an effective CPU mask:

Server1 ID1EAX (0x00000F43)  0000 0000 0000 0000 0000 1111 0100 0011
Server2 ID1EAX (0x00000F25)  0000 0000 0000 0000 0000 1111 0010 0101
Server3 ID1EAX (0x000006F6)  0000 0000 0000 0000 0000 0110 1111 0110
Standard CPU Mask ID1EAX     XXXX HHHH HHHH XXXX XXXX HHHH XXXX XXXX
Custom CPU Mask ID1EAX       ---- ---- ---- ---- ---- 0--0 ---- ----
Effective CPU Mask ID1EAX    XXXX HHHH HHHH XXXX XXXX 0HH0 XXXX XXXX

Based on the information in this Intel documentation, this register (ID 1 EAX) is the CPU identification register, and masking these bits will make it difficult (or perhaps even impossible) for ESX Server (or VirtualCenter) to correctly identify the CPUs in your host servers.  This is the underlying reason these kinds of changes aren’t supported:  no one really knows what impact this will have.

This is just the ID1EAX register, though; we must now repeat this process for the other registers:

Server1 ID1ECX (0x0000641D)  0000 0000 0000 0000 0110 0100 0001 1101
Server2 ID1ECX (0x00004400)  0000 0000 0000 0000 0100 0100 0000 0000
Server3 ID1ECX (0x0004E3BD)  0000 0000 0000 0100 1110 0011 1011 1101
Standard CPU Mask ID1ECX     RRRR RRRR RRRR RRR0 00XR R0H0 000H 0RRH
Custom CPU Mask ID1ECX       ---- ---- ---- -0-- ---- --0- ---0 -0-0
Effective CPU Mask ID1ECX    RRRR RRRR RRRR RRR0 00XR R000 0000 00R0

Server1 ID81ECX (0x00000000) 0000 0000 0000 0000 0000 0000 0000 0000
Server2 ID81ECX (0x00000000) 0000 0000 0000 0000 0000 0000 0000 0000
Server3 ID81ECX (0x00000000) 0000 0000 0000 0000 0000 0000 0000 0001
Standard CPU Mask ID81ECX    RRRR RRRR RRRR RRRR RRRR RRRR RRRR RRRX

Server1 ID81EDX (0x20000000) 0010 0000 0000 0000 0000 0000 0000 0000
Server2 ID81EDX (0x00000000) 0000 0000 0000 0000 0000 0000 0000 0000
Server3 ID81EDX (0x20100000) 0010 0000 0001 0000 0000 0000 0000 0000
Standard CPU Mask ID81EDX    RRXR RRRR RRRH RRRR RRRR HRRR RRRR RRRR
Custom CPU Mask ID81EDX      --0- ---- ---0 ---- ---- ---- ---- ----
Effective CPU Mask ID81EDX   RR0R RRRR RRR0 RRRR RRRR HRRR RRRR RRRR

You’ll note that there is no custom mask for ID81ECX; that’s because the default CPU mask hides all the significant bits (which in this case is only bit 0).

With these changes in place, I was able to successfully VMotion several different guest operating systems (including OpenBSD 4.1, Solaris 10 x86 Update 3, and Windows Server 2003 R2) between the servers.  Your mileage may vary, however, and keep in mind that these changes are unsupported.  Use them at your own risk!  (I have to say that because if I didn’t someone would invariably blame me for making one of their guest VMs crash.  With that disclaimer out of the way, allow me to state that I haven’t yet seen any problems as a result of the custom CPU masks.)

As always, please feel free to add any thoughts or corrections in the comments below.

The rumor that a slimmed down version of ESX Server, supposedly called “ESX Lite,” is being developed for placement into a server’s firmware, is circulating the Internet (see here or here).  Most of these reports link back to a story on SearchServerVirtualization which quotes sources close to VMware stating:

According to several sources close to VMware, ESX Lite is real and currently under development. The new lightweight hypervisor would be installed directly on the motherboard, simplifying the deployment of an ESX host and ensuring 100% hardware integration.

Virtualization.info adds that the rumor is apparently confirmed.

This is a smart move.  It’s smart because it derails Microsoft’s attempts to marginalize the hypervisor by bundling it with the operating system (via Windows Server Virtualization, aka “Viridian”).  It’s smart because it expands the hypervisor market in new directions that no one else has yet tapped, helping VMware retain mindshare about its technical leadership and innovation.  It’s smart because it’s the hardware vendors that have the most to lose via virtualization, and by partnering with them you remove potential future opponents.

However, it’s also a very risky move.  What if “ESX Lite” doesn’t (or can’t) perform as well as “full” ESX Server?  What if embedding the hypervisor into a server’s firmware causes VMware to lose visibility?  After all, customers wouldn’t be buying solutions from VMware any longer, because these would be integrated with the hardware.  It would be prudent for VMware to create a kind of “VMware Inside”-type program that maintains their visibility in the overall solution, even though it’s all being purchased through Dell, HP, or IBM.

It will be very interesting to see how this plays out, and which of the hardware vendors are “on board” with the effort.

UPDATE:  In a recent blog posting, Gordon Haff agrees with me regarding this approach by VMware as a move to forestall the “built in” virtualization functionality that exists or will soon exist in most operating systems.

This IBM BladeCenter H installation I’ve been working on with another engineer for the last couple of days is not going as smoothly as we both would have liked.  I don’t know if this is indicative of the BladeCenter H chassis itself, or if it’s just me.  While some would say it’s just me, I suspect it’s a little of both.

For example, we ran into issues with the Management Modules “freaking out,” and failing over between the modules didn’t solve the problem.  In fact, we had to power down both management modules and then power them back up one at a time in order to get the fans to finally settle down.  Otherwise, the chassis sounded like a jet turbine getting ready to take off, and—get this—people outside the datacenter could hear the chassis.

Even after getting that settled down, there was still some weirdness with the IBM KVM switch that we still haven’t resolved (kept freaking out the keyboard and causing it to stop working).  We had to plug in a separate USB keyboard in order to work with the chassis.  That particular issue still hasn’t been resolved.

The real kicker, though, was the problem we ran into with the floppy drive.  The design called for boot from SAN via iSCSI (using LUNs off a Network Appliance storage system) using Qlogic HBAs.  This is a supported configuration, but requires the use of a driver floppy during the installation of Windows Server 2003.  The older BladeCenter chassis had a built-in floppy, but the BladeCenter H does not, and the USB floppy that we tried to use wouldn’t work.  No matter how hard we tried, the blade(s) just wouldn’t see the floppy drive.  Until we can get a floppy drive recognized during the Windows setup process, we can’t get Windows installed and we are just stuck.

Anyone else run into similar issues with a BladeCenter H?

While I would love to be able to say that this procedure I’m going to describe for resizing disks inside guest VMs is new or unique or original, I can’t.  I’m sure that lots of smart people out there have been down this path before, and more than a couple of them have probably written up good instructions on how to do it.  I’m including this information here partly for myself (in the event I need it in the future), and partly because the information fits in with a lot of the other information I have here on VMware and related technologies.

So, with that being said, here’s how I recently went about resizing some guest VM disks using vmkfstools and GParted (specifically, the GParted LiveCD).  This process assumes you need to resize a Windows guest.  The process should be very similar if not the same for non-Windows guests, but I haven’t tested it so I can’t be absolutely certain.  The process is very straightforward and not unusual in the least, but feel free to post any corrections or questions in the comments.

1. Download the GParted LiveCD (here’s a direct link) and then upload it to your ESX Server using the file upload tool of your choice.  Being a Mac OS X user, I use Interarchy.  Use whatever best suits you.
2. Power off the VM that you will be manipulating.  (I know, that seems obvious, but still…)
3. Open an SSH session to the ESX Server where your guest VM is hosted.  Switch to root, change into the appropriate directory where the VM is stored, and then run the following command:

   vmkfstools -X <New disk size> <VMDK filename>

   If your current virtual hard disk was 10GB and you wanted to make it 20GB, then specify “20g” on the command line, i.e., specify the new total size of the disk, not the amount by which to increase it.

4. Flipping to the graphical VI Client, change the CD-ROM of the guest VM to be the ISO image of the GParted LiveCD you downloaded earlier.  Make sure it is set to connect at power on.  Power on the VM and boot from the CD.

   Note that booting from the CD can be a bit tricky.  You may need to boot up several times before you catch it just right.  Be sure that if Windows starts booting up, you let it boot up and then shut it back down again.  If you reset the VM in the midst of the Windows boot sequence, the NTFS filesystem will be marked as “dirty” and GParted won’t make the changes you wanted.

5. GParted will boot up from the CD.  You may need to press Enter a couple of times to accept the defaults (unless you need settings other than the defaults, of course) before the graphical environment loads and you see the GParted interface.  Once the GParted interface is up, you should be able to figure out how to make the changes you want to make.
6. Shutdown the VM (you can use the shutdown option in GParted), disconnect the CD-ROM, and power the VM back up again.  When Windows boots, it will run a Chkdsk, then reboot again, and then come up to a login screen.  After you login, you may be prompted to reboot again after the discovery of “new hardware.”  After that final reboot, you should be good to go with a new, larger virtual hard disk.

The most time consuming portion, in my experience, is waiting for GParted to boot from the ISO image.  Otherwise, the entire process is almost completely painless.