blog.scottlowe.org

I purposefully scheduled a later flight out of Atlanta because I wanted to allow plenty of time to get from Duluth (where my training class was located) to the airport.  Knowing that traffic in Atlanta can be troublesome at times (cough, cough), I wanted to be sure to allow plenty of time.

Instead, my class ends early and I think, “Well, I’ll just head on down to the airport and get an earlier flight.”  Hah!  Right!  No, sorry—no seats on any earlier flights available.  Oh, and by the way, I can’t check in to my original flight yet because (get this) I’m too early.

So I waste a few hours until I can get checked in, then need to waste a few more hours before my flight is scheduled to leave.  As it turns out, there are numerous storms, snowstorms, bad weather, etc., in the Midwest, and the resulting chaos/delays/cancellations is overflowing into Atlanta.  Oh, joy!  Rumor has it that the airlines are overbooked and offering to put people up overnight in exchange for their seats.  A lady in the line with me to check in says she’s been delayed three times and has now been bumped to an entirely different airline.

This smells like a serious delay to me, and I ought to know what a delay looks and smells like—I’ve certainly seen my share.  I’m beginning to believe that the comment a co-worker in my class made is true: “Man, you have bad luck with flights.”  Tell me about it!

Next time I have to come to Atlanta, I’m renting a car and spending the eight hours on the road instead of sitting in the airport.

I haven’t had the chance to finish installing Solaris 10 yet, as I ran out of time trying to copy down the ISO images before I left to go to Atlanta for some Network Appliance training.  (By the way, have I mentioned how cool NetApp’s stuff is?  No?  It is.)

After first running into the problem with the Solaris 10 VM, I inquired with other private beta testers and later filed a bug report.  Another user suggested I check this KB article; I tried adding the monitor_control.disable_longmode parameter to the VMX file and….it worked!  Where Solaris had previously attempted to load a 64-bit kernel (and caused a host kernel panic) it now loaded a 32-bit kernel and everything seemed to work just fine.

I hope to finish the Solaris installation within the next couple of days and continue on with some additional testing.

UPDATE:  I finished the Solaris installation earlier today and it runs just fine.  I haven’t installed the VMware Tools for Solaris yet, but based on my work with Solaris on ESX Server I don’t expect any problems there.

The article at virtualizationdaily.com is pretty cool, but I’d be much more interested if it were written the other way around—converting a Parallels VM to run under VMware.  Of course, that’s not likely to happen until VMware Fusion hits public beta.  If I had a copy of Parallels (or a Parallels VM), I’d try it myself.  Anyone out there with a Parallels VM they want to donate?  Hey, wait a minute…I could just try one of the prebuilt Parallels appliances also made available at virtualizationdaily.com!  Stay tuned for more details….

In other VM portability news, I had the opportunity today to try moving a VMware created with VMware Workstation 5.5.x on Windows to VMware Fusion on my MacBook Pro.  A colleague of mine had built a Red Hat Linux VM with the Network Appliance Data ONTAP Simulator, and since I’m out of town in a Network Appliance boot camp, I thought it would be handy to have a copy of that.  After all, it would save me the time of installing Red Hat, etc.  A quick copy onto a USB 2.0 hard drive and then onto my MBP and I was ready to roll.  The VM needed very little tweaking to run; I only had to force NAT networking for both of the VM’s network interfaces and everything ran just fine.  That bodes extremely well for the future of VMware Fusion and its compatibility (or integration) with VMware Workstation on other host operating systems.  As VMware continues to develop and polish this software, it will only get better—and it’s not half-bad right now (even for beta software).  Today, for example, I had both a Red Hat Linux VM and a Windows XP Professional VM up and running and the laptop still didn’t even seem to be breaking a sweat.  Cool!

The next trick will be to take a VM created with VMware Fusion and move it over to VMware Workstation on Windows or Linux.  If I get a chance to try that, I’ll post more information here.

A number of the readers of my article describing integration between Linux and Active Directory on Windows Server 2003 R2 have inquired about the need to install Server for NIS on a domain controller.  Even though we don’t necessarily need NIS for this process (although we will need NIS if we are going to use NFS and automounts), installing the Server for NIS also makes available the “UNIX Attributes” tab in the Active Directory Users and Computers console.  You’ll need some sort of access to the attributes in Active Directory (unixHomeDirectory, gidNumber, uid, uidNumber, gecos, loginShell) in order to set them so that Linux and UNIX systems can utilize the information in those attributes, so installing Server for NIS in order to get the “UNIX Attributes” tab makes sense.

It’s not the “UNIX Attributes” tab that’s important; it’s access to those attributes in Active Directory.  You could just as well use ADSI Edit, LDP, or programmatically edit the attributes via VBScript or an LDIF import file.  It doesn’t matter.  All that matters is that you have the ability to set and modify the values in the UNIX-related attributes.

One common workaround that has been mentioned is just registering the nisprop.dll file, using a command like this:

regsvr32 c:windowsidmucommonnisprop.dll

Normally, this trick would work well.  I used this trick, for example, to make Active Directory Users and Computers available to help desk personnel without having to install all the administrative tools (just copy down dsadmin.dll and register it).  Not this time, though.

As Andy Loggia pointed out to me (first in the comments, and again later in a separate e-mail message), registering nisprop.dll requires Schema Admin privileges.  At first, I didn’t believe him, but he’s absolutely right.  When you register nisprop.dll, a change needs to be made in the Configuration naming context of Active Directory—and making that change requires Schema Admin privileges.

Specifically, registering nisprop.dll adds the CLSID of nisprop.dll to the AdminPropertyPages attribute of the user-display and group-display objects in this location in Active Directory:

CN=409,CN=DisplaySpecifiers,CN=Configuration,DC=example,DC=net

(The “CN=409” would change if you are running a language other than English.)  I verified this myself on my own instance of Active Directory in the lab and Andy is absolutely correct.  Good work, Andy!

If you are working on Linux-AD integration in your shop, then just keep in mind that at some point during the process you’ll probably need to have Schema Admin privileges.  Certainly while you are extending the schema (if it’s not already extended, which you can check using ADSI Edit), then when you install Server for NIS or register nisprop.dll.  Alternately, if you don’t want the “UNIX Attributes” tab in Active Directory Users and Computers, you can use tools such as LDP, ADSI Edit, LDIF import files, or scripts to populate and edit the values in the UNIX-related attributes.  Populating these values is necessary for the process to work correctly, but the method by which the attributes are populated is up to you.

(Thanks to virtualization.info for alerting us to this release.)  VMware has posted an update to VirtualCenter that addresses a potential security problem with the Virtual Infrastructure clients and the VirtualCenter server.  Apparently, the potential security flaw is that SSL server certificate validation was not being performed, allowing a potential man-in-the-middle attack.  The update enables the validation of SSL certificates; the exact procedure for how this is done is documented in this VMware KB article.

Richard Garsthagen of www.run-virtual.com has published an application (I believe he’s calling it VMotion Info) that clearly identifies the differences between CPUs across servers in a farm of ESX servers.  This application talks to the VirtualCenter server and then gather information from all the ESX servers represented in VirtualCenter.  After gathering all the information, it will then graphically present the differences and actually decode the raw bits to tell you exactly what the differences are (i.e., NX/XD supported/not supported, SSE2/SSE3, etc.).

This is extremely useful information.  The missing piece, unfortunately, is the conversion to the binary bits that are necessary to actually mask that information from the virtual machine.  I provided some of that information in my earlier article, but for completeness’ sake thought I might present it here again.  Hopefully, this will prove helpful to someone.

For example, here are the raw binary values I obtained on two servers in my server farm:

ID1ECX 0x0000641d (server one)
ID1ECX 0×00004400 (server two)

Converted to binary, that becomes:

0000 0000 0000 0000 0110 0100 0001 1101 (server one)
0000 0000 0000 0000 0100 0100 0000 0000 (server two)

Observing the binary values, we can easily see the differences between the two CPU families and why (by default) VMotion won’t work across them.  But what do these values mean?  And how does this information play into Richard’s VMotion Info tool?  Good questions.

Taken from this Intel document, here are the meanings of those binary values (bit 0 would be the rightmost bit in the binary values above):

0 - SSE3
1 and 2 - Reserved
3 - MONITOR
4 - DS-CPL
5 - VMX (Intel Virtualization Technology)
6 - Reserved
7 - EST (Enhanced Intel SpeedStep Technology)
8 - TM2
9 - SSSE3 (Supplemental SSE3)
10 - CID
11 and 12 - Reserved
13 - CX16
14 - xTPR
15 through 17 - Reserved
18 - DCA (Direct Cache Access)
19 through 31 - Reserved

In the example provided above, the differences between the CPUs were in bits 0, 2, 3, 4, 10, and 13. That translates into differences in SSE3, MONITOR, DS-CPL, CID, and CX16 support between the two CPUs.

However, not all those features are necessarily significant to VMotion.  In order to know what features are significant to VMotion, we’ll need to review the default CPU mask that’s provided by VirtualCenter to new virtual machines.  Based on my research, the default mask provided to a new Windows Server 2003-based virtual machine is this:

RRRR RRRR RRRR RRR0 00XR R0H0 000H 0RRH

According to the legend from VirtualCenter, any bit marked R or H must match for successful VMotion.  This means that although bits 0, 2, 3, 4, 10, and 13 are different, only bits 0, 2, 4, and 10 are actually significant to VMotion (bits 3 and 13 are already marked 0 [Hide feature from guest software] or X [Unused by guest software]).  In order to get VMotion working, we’ll need to mask those bits that are different and are significant to VMotion compatibility.

In addition, there are multiple registers—ECX and EDX—and different values (0, 1, 80000000h, and 80000001h).

I’m assuming that Richard’s tool only identifies those flags that actually impair VMotion compatibility.  Those flags, and their corresponding register bits are listed below:

NX/XD - Level 80000001 EDX bit 20
FFXSR - (unknown)
RDTSCP - (unknown)
SSE - Level 1 EDX bit 25
SSE2 - Level 1 EDX bit 26
SSE3 - Level 1 ECX bit 0
SSSE3 - Level 1 ECX bit 9
CMPXCHG16B - Level 1 ECX bit 13

I couldn’t find the feature flag values for FFXSR and RDTSCP in the Intel documentation.  Not shown above (as far as I can tell) is the flag for the Intel 64-bit extensions, which is found at Level 80000001 EDX bit 29.

So, if after running the Virtual Info tool you find differences in NX/XD and Intel 64 support, then you can create a custom CPU mask that hides (using 0 in the mask) level 80000001 EDX bit 20 and level 80000001 EDX bit 29.  By placing a zero in the mask, then these values are hidden from the VM and therefore won’t affect VMotion compatibility.

Using the Virtual Info tool, you can now easily and clearly identify the differences in your CPUs.  Then, using the breakdown listed above, you can easily create the necessary CPU masks to hide those differences and increase VMotion compatibility across hosts.  Keep in mind, though, that most of the CPU masks are currently unsupported by VMware.  (I just noticed that the supported vs. unsupported “relaxations,” or masks, are listed at the bottom of Richard’s tool.)

Also, there’s an ongoing VMTN forum discussion occurring about CPU compatibility and VMotion; Mike Laverick (of RTFM Education) also mentioned it on his weblog as well (see the section marked “New Community Initiatives”).

This solution builds upon the integration of Solaris 10 and Linux (again, CentOS specifically, but I’ll use Linux here instead of mentioning the specific distribution) into Active Directory for authentication and authorization as outlined in these related articles:

Refined Solaris 10-AD Integration Instructions
Linux, Active Directory, and Windows Server 2003 R2 Revisited

In these articles, we describe a configuration whereby we can use Kerberos against Active Directory for authentication, and LDAP against Active Directory for user and group lookups.  This allows us to centralize the user account information (username, UID number, home directory, login shell, etc.) into Active Directory so that when an account is provisioned in AD it is also available to associated UNIX and Linux systems (as applicable).  Using the information in this article, we can extend that integration to auto-mounted shared home directories between Solaris and Linux.

We’ll accomplish this by using the UNIX interoperability tools included in Windows Server 2003 R2; namely, the Server for NFS, Server for NIS, and the User Name Mapping Server.  On the Solaris and Linux side, we’ll use autofs.

OK, let’s dive right in.

Configuring Solaris

Configuring Solaris is pretty straightforward; it’s pretty much designed out of the box to use auto-mounted home directories.  This is one area where Solaris shows its enterprise pedigree (that’s not a knock against Linux, just an observation about Solaris).

The only real change that needs to be made is to the /etc/auto_home file, which specifies the auto-mounted home directory mappings.  The easiest thing to do is use a wildcard mapping, like this:

*   nfssrvr.example.com:/home/&

Here, the asterisk (*) and the ampersand (&) will be replaced with the appropriate information.  So, if a user name “bjones” logs on, then the the /home/bjones NFS share on the server nfssrvr.example.com would be mounted on /home/bjones on the Solaris server.  Useful, eh?

Note that autofs may be disabled in your installation of Solaris; it wasn’t in mine, but I’ve seen references elsewhere that this is not typical.  Use these commands to check the status and enable autofs, if necessary:

svcs -a | grep autofs
svcadm -v enable svc:/system/filesystem/autofs

At this point, Solaris should be ready to go.  Let’s look at the Linux configuration.

Configuring Linux

The Linux configuration is only slightly more complicated than the Solaris configuration.  Whereas Solaris uses /etc/auto_master to define the automounted filesystems, autofs on Linux uses /etc/auto.master, and it does not (by default, as far as I can tell) define a map for automounted home directories.

So we must first edit /etc/auto.master and add the following line:

/home   /etc/auto.home

This tells autofs to look in the file /etc/auto.home to find the automounted directories under /home.  Then we create the /etc/auto.home file with the following line:

*   nfssrvr.example.com:/home/&

(Just like the Solaris box in this case.)  Finally, we can check for the autofs startup status, fix it, and start autofs with the following commands, respectively:

chkconfig --list autofs
chkconfig autofs on
service autofs start

In fact, you will need to restart the autofs daemon (with “service autofs restart”) in order for the changes to /etc/auto.master and /etc/auto.home to be recognized.  (The same goes for Solaris, too, only using the “svcadm -v restart svc:/system/filesystem/autofs” command instead.)

To further streamline any differences between Solaris and Linux, I also created a /export/home on the Linux server, in the event I wanted/needed local home directories as well.  Please note that I would need to modify the /etc/auto.home (on Linux) or /etc/auto_home (on Solaris) as well if this were the case.

Now that Linux and Solaris are both done, let’s have a look at configuring the UNIX interoperability components on Windows.

Configuring Windows (NFS, NIS, and User Name Mapping)

Surprisingly enough, this part was the hardest part for me.  I don’t know if it was the documentation (if it can be called that) or something else, but I had a hard time with this.  Hopefully the information I share here can help someone else avoid this headache.

Configuring the Domain Controllers

On the domain controllers in your Active Directory domain, install the following components:

• Server for NIS and Administration Components (under Active Directory Services > Identity Management for UNIX)
• Microsoft Services for NFS Administration, RPC External Data Representation, RPC Port Mapper, Server for NFS Authentication, and User Name Mapping (under Other Network File and Print Services > Microsoft Services for NFS)

It may be possible to get by with less than that installed, but I couldn’t get it working until all of those are installed.  If you’ve followed my instructions for AD integration, the Server for NIS and Administration Components are probably already installed, as those expose the “UNIX Attributes” tab in Active Directory Users & Computers.

Once these are installed, use these steps to configure them appropriately:

1. Using the Services MMC, make sure that Server for NIS is running.
2. In the Microsoft Services for NFS MMC, right-click on “Microsoft Services for NFS” and select Properties.  Specify the name of the DC as the “User Name Mapping Server”, check the box labeled “Active Directory Lookup”, and specify the name of the Active Directory domain.  Click OK to save those changes and close the dialog box.
3. Still in Microsoft Services for NFS MMC console, right-click User Name Mapping and select Properties.  Go to the “Simple Mapping” tab, check the box marked “Use simple maps”, and then add one of the DCs from the domain that’s running Server for NIS at the bottom of the dialog box.
4. Right-click on both User Maps and Group Maps and select “Show simple maps,” then click Refresh.  You should see a list of the users in Active Directory that have been UNIX-enabled, i.e., have UNIX attributes specified for their account.  At this point, you can close the Microsoft Services for NFS MMC.

The DC should now be ready to go.  It’s time to move to the file server.

Configuring the File Server

We need to install the Server for NFS component on the file server that will host our automounted home directories.  The Server for NFS is found under “Other Network File and Print Services” in Add Windows Components.

Once that has been installed, repeat step #2 above (configuring the User Name Mapping Server and Active Directory lookup) on the file server.

We’re now ready to create some NFS shares.  Right-click the folder you’d like to share via NFS (I created a folder named “home”—I know, not very creative) and shared that via NFS (there’s an NFS Sharing tab when you open the properties for the folder).  When specifying the share name, be wary of the case since UNIX/Linux are cAsE sEnSiTiVe.  (It’s probably best to stick to all lowercase.)  If you want to allow anonymous access and/or root access, configure it appropriately at this point.

Word of warning:  be sure to include UNIX-enabled groups and/or users in the NTFS access control list for the folder and containing files!  This one threw me for a loop.  At first, I didn’t have any UNIX-enabled groups in the ACL and I kept getting “insufficient permissions” errors when trying to mount the share via NFS.  It wasn’t until I ensured that at least one UNIX-enabled group had read permission to the folder that things started working as expected.

To test your NFS share, use this command from Solaris or Linux, respectively:

mount -F nfs nfssrvr.example.com:/home/nfs /mnt/nfs (for Solaris)
mount -t nfs nfssrvr.example.com:/home/nfs /mnt/nfs (for Linux)

If this mounts the NFS share without any problems, then you should be ready to roll.

(Note:  In case you hadn’t already picked this up, one advantage of using a Windows NFS server is that we can also share the same folder via CIFS/SMB and create unified home directories for Solaris, Linux, and Windows.)

Now, upon logging in to either Solaris or Linux (I tested this via SSH), you should get dropped into “/home/username” (assuming that’s where you configured the automounter to mount the NFS share) automatically.

How I Tested

I tested this procedure using Solaris 10 6/06 and CentOS 4.3.  These systems were configured for Active Directory integration and native Kerberos logons via SSH.  I used Windows Server 2003 R2 Standard Edition for the NFS server, and Windows Server 2003 R2 Standard Edition for the Active Directory domain controllers.  All of these systems were created as virtual machines hosted on a VMware Virtual Infrastructure 3 server farm with servers running ESX Server 3.0.1.

Using VMware Fusion (or I suppose that other Intel-based Mac virtualization application, Parallels Desktop), we can now run a Windows instance on our Mac (I’m running it on a Core 2 Duo-based MacBook Pro, for example).  Like all other VMware hosted products (VMware Workstation or VMware Server, both on either Windows or Linux), the close-enough-to-native performance allows us to use this instance in order to host an installation of the Virtual Infrastructure (VI) client without an unacceptable performance impact.

OK, that’s pretty straightforward, and not too complicated.  But what happens when you open a VI console to a virtual machine hosted on your server inside this VM?  Then you’ve got a virtual machine console running inside a virtual machine.

The cool thing about this is not only that it works, and works just fine, but that the software is intelligent enough to recognize the layers of abstraction.  For example, today I was running a copy of Windows XP Professional under VMware Fusion, and inside that instance of Windows I had the VI client running.  With the VI client, I opened a virtual console to a Linux guest running on my ESX server farm.  In order to escape out of the Linux console to the Windows VM, I had to press Ctrl-Alt.  But that didn’t release the VMware Fusion console, which required that I press Ctrl-Cmd.  (Please note that this may not work so well when the operating systems are the same in both the guest and the host.)

Of course, this isn’t the only workaround for the lack of Mac support in managing ESX server farms; I’ve heard of people using Remote Desktop Connection (or a similar RDP client) to connect to a Windows instance running on the farm itself.  I’ve tried that, too, but haven’t been too pleased with the performance of the VM consoles inside an RDP session.  I haven’t tested Citrix yet, but with the recent release of a free 2-user Citrix license I plan to remedy that shortly.  I’ve also read accounts (most notably from Rui Carmo’s Tao of Mac) of people using X11 to display the consoles on a Mac (even though they are executing elsewhere).  Again, I’ve not tested that so I can’t testify as to the performance.  I can say that the performance of using a locally hosted VM to access the VI client has been good thus far.

What about you other Mac users out there?  What workarounds have you employed (or are employing right now)?  I’d love to hear some of your ideas.