blog.scottlowe.org

I’m in the midst of a non-virtualization-related project right now, which is a bit odd; a great majority of my work these days is centered around virtualization.  Nevertheless, I try to view every project as one from which I can learn.  I have definitely learned some things with this project, that’s for sure.

Here are a few tidbits that I’ve learned so far this week, most of them centered around the installation of Microsoft Exchange Server 2007:

• First, if you have even one Active Directory domain controller that isn’t running Windows Server 2003 SP1, you can’t use the GUI setup routine for installing Exchange Server 2007.  That’s right, no GUI setup for you.  Instead, you’ll have to install from the command line like this:

  setup.com /mode:install /roles:mb, ca, ht, mt /EnableLegacyOutlook /LegacyRoutingServer:oldserver.domain.com /dc:win2k3sp1.domain.com

  Nice, eh?  Supposedly this will be fixed in Exchange 2007 SP1.

• Apparently, about 20% of the installations run from the command-line fail with an error about being unable to access the source files.  This is even when installing from local CD-ROM, as I was.  The Microsoft tech I spoke with recognized that this was a problem; the suggested solution was to copy the files from the CD to a local hard drive and run setup from there.
• The use of the “/LegacyRoutingServer:” command-line switch, which is required for interoperability with “legacy” Exchange 2000/2003 servers, can only be used when installing the very first server with the Hub Transport role (the “ht” in the command line above).  If the installation of that first server dies for some reason—say, like due to some strange error about not being able to access the source files—then you won’t be able to use this command-line switch again.  This means you’ll need to create the appropriate connector yourself manually after installation.
• If Exchange Server 2007 setup fails while installing the Client Access server role (the “ca” in the setup command line above) citing an error about not being able to find an object (see this URL), then you’ve got some damaged attributes in Active Directory.  In my case, while sitting on hold with Microsoft Support for an hour, I resolved it by doing a full dump of the domain and configuration naming contexts of Active Directory using LDIFDE:

  ldifde -f example.domain.ldf -d “dc=example,dc=com”
ldifde -f example.config.ldf -d “cn=configuration,dc=example,dc=com”

  I was then able to find the specific object with which Exchange Server 2007 Setup was having a problem and fix it.  In my case, the Offline Address Book server had somehow gotten damaged and was causing setup to fail.  I was able to manually correct the problem using Exchange 2000 System Manager and then Exchange Server 2007 setup proceeded to completion.

• Specifying a smart host on the SMTP virtual server properties on your “legacy” Exchange servers will cause a routing loop, and mail won’t flow between the new and old servers.  Apparently this is documented somewhere, although the Microsoft tech I spoke to could only point me to some articles about how to configure a smart host.  I haven’t seen any documentation yet that recommends checking and fixing this potential problem.  Furthermore, the troubleshooting tools in Exchange Server 2007 pick this up, but fail to tell you that it could be a problem.
• Oh, yes, I almost forgot about one: ASP.NET is required for Exchange Server 2007, but what happens when you can’t install it via Add/Remove Programs > Add/Remove Windows Components?  That’s right, back to the command line again:

  %systemroot%\Microsoft.NET\Framework64\
v2.0.50727\aspnet_regiis.exe -ir -enable

  This is assuming, of course, that you’ve already installed the .NET Framework 2.0 on your server in preparation for Exchange.

You are welcome to tell me that I’m an idiot for not knowing this stuff, on one condition: you provide a URL where information about the problem is posted and a workaround provided.  That way, when someone else runs into the issue, we’ll at least know where to point them for help.

I won’t bore you with all the details again, since no doubt by now you’ve probably already seen the news—or perhaps I should say hype—about how the End-User License Agreement (EULA) for Leopard Server (Mac OS X Server 10.5) has changed to apparently allow for virtualization of Mac OS X Server.

Quoting the EULA from the originating article:

This License allows you to install and use one copy of the Mac OS X Server software (the “Mac OS X Server Software”) on a single Apple-labeled computer. You may also install and use other copies of Mac OS X Server Software on the same Apple-labeled computer, provided that you acquire an individual and valid license from Apple for each of these other copies of Mac OS X Server Software.

Now, this isn’t quite what I had discussed in my last post about Apple and virtualization, but it is a step in the right direction.  Unfortunately, the current EULA limits these virtual instances to be a) version 10.5 only; and b) Mac OS X Server only.  Alas, no virtual instances of “regular” Mac OS X for geeks such as myself, and no earlier instances of Mac OS X either.

Of course, the other big question surrounds who will win the race to produce the first application to provide virtualized Mac OS X Server instances…will it be VMware or Parallels?  To be honest, I’m inclined to say Parallels, but maybe Ben, Regis, and others at VMware can prove me wrong.

Here are a few other links with related information:

http://www.tuaw.com/2007/10/31/will-leopard-allow-virtualization-of-os-x-server/

http://www.macnn.com/articles/07/10/31/os.x.server.on.vm/

http://apple.slashdot.org/article.pl?sid=07/10/31/1629236

Quite a few organizations implementing VMware Virtual Infrastructure 3 (VI3) are not organizations with a great deal of non-Windows experience.  Given that the Service Console is based upon Red Hat Enterprise Linux, this is often a completely new environment for them that introduces unfamiliar processes and applications.

Take the process of copying files, for example.  To an administrator within an organization that is primarily Windows-based servers, the idea of not being able to “map a drive” or just use a Universal Naming Convention (UNC) path is unbelievable.  For someone like myself, who has lived in a non-Windows world for a number of years now, the idea of using SCP/SFTP to copy files to a server seems pretty normal.  Nevertheless, these new VMware administrators must now get accustomed to doing things a bit differently than they have in the past, and the use of an SCP/SFTP client is one of those things.

One big problem with the use of SCP/SFTP clients is granting SCP/SFTP access to a user typically also grants shell access.  This is a problem not only for new users but also for experienced users.  New users may get into the shell and change things that shouldn’t be changed, simply because they don’t know; experienced users may get into the shell and change things that shouldn’t be changed, because they think they do know.  Either way, shell access is something to be avoided where possible, in my opinion.

The solution?  Use an “alternative” shell called scponly:

scponly is an alternative ’shell’ (of sorts) for system administrators who would like to provide access to remote users to both read and write local files without providing any remote execution priviledges. Functionally, it is best described as a wrapper to the tried and true ssh suite of applications.

Here’s how I got scponly working on a test server running ESX Server 3.0.1 in the lab.  Please keep in mind that this method is most likely not supported by VMware, so proceed accordingly.

1. Download the scponly and rsync RPM packages from the DAG repository.  (The links point to the package page, not the actual package itself, so these are not blind download links.)  I used scponly-4.6-3.el3.rf.i386.rpm and rsync-2.6.8-1.el3.rf.i386.rpm, the packages for Red Hat Enterprise Linux 3, since the Service Console is derived from RHEL 3.
2. As root, first install the rsync package with the rpm command.  Something like this should work:

   rpm -Uvh rsync-2.6.8-1.el3.rf.i386.rpm

3. After installing rsync, install scponly:

   rpm -Uvh scponly-4.6-3.el3.rf.i386.rpm

4. Edit the /etc/shells command to include the scponly shell, which in my testing was installed to /usr/bin.  You can simply append /usr/bin/scponly to the end of /etc/shells.

At this point, you should be able to create a user and set the user’s shell to scponly.  This will allow the user to use an SCP/SFTP client to transfer files, but it will not allow interactive shell access.

To test this, I used three file transfer methods:

• Interarchy, a Mac OS X application, via the SFTP protocol
• Command-line SCP from my Mac
• WinSCP on Windows XP, via both SFTP and SCP

In all three cases, I was able to successfully authenticate and transfer files, but I was not able to get access to the shell via SSH.  I would imagine that a determined individual could probably get by scponly, but in this instance we are only using it to provide a layer of protection for the shell.  For that function, it works well.

The Center for Internet Security (CIS) has released some security benchmarks for VMware ESX Server 3.0.x.  The ESX security benchmark joins recommendations and guidelines for Windows 2000, Windows XP, Windows Server 2003, Red Hat Linux, and Mac OS X that are also available from the CIS.  The CIS has also published a generic virtual machine (VM) security benchmark as well.  Taken together, the ESX benchmark and the VM benchmark provide good, solid recommendations around virtualization security.

The ESX/VM benchmarks are available for download here.

With all the hype around virtualization’s impact on security—positive or negative—it’s good to see some concrete and actionable recommendations.  If nothing else, these documents will at least provide a starting point for security professionals and virtualization experts to discuss the best way to architect solutions without compromising the security of the network/servers.  In fact, we might even find ways to enhance the security of the network/servers.

I feel it’s also important to mention that a number of the recommendations found in the CIS benchmark are also included in the VI3 Server Configuration Guide (available here from VMware’s web site).  Why is this important to know?  In my opinion, VMware has done a reasonable job of making security information available.  They haven’t done a stellar job, but they haven’t done a shoddy job either, and I believe that many of the concerns around “virtualization is insecure” derive directly from the failure of implementing parties to properly apply the knowledge that was available from the vendor.  That means reseller/VAR SEs, like myself, who help customers implement this stuff are the ones responsible for making sure that we know what VMware’s recommendations are and that we properly relay those to the customer(s).  Of course, if the customer chooses not to follow recommendations, that’s another story…

In any case, go get the CIS benchmarks, review them, and put them to work.

Thanks to Chris Hoff for alerting everyone to the release of the ESX Server benchmark.

I used to read Tom Yager’s “Ahead of the Curve” column when InfoWorld was still a print publication.  Every month, without fail, as soon as the magazine arrived I turned right to his column.  It was one of my primary reasons for reading the magazine, at least in recent years.  I think it’s probably safe to say that Yager’s affection for Mac OS X led me to perform an evaluation of my own and, eventually, to switch to Mac OS X myself based on the results of that personal evaluation.

But then the magazine turned digital/online only, and I stopped following his column.  I already had enough stuff coming in to my various digital inboxes, and didn’t really need another.  Part of the allure of the column had been precisely that it wasn’t digital.

Fast forward to just the other day, when I stumble across his column once more and find that I, too, am “ahead of the curve.”  In his recent article The next best thing to OS X, Yager claims that Sun Solaris 10 is a great fit for places where Mac OS X isn’t.

Given that I have embarked upon a plan to learn Solaris, it’s kind of nice to see an “industry analyst” say that you’re making the right move and that you, too, are ahead of the curve.

I’m not really sure how to respond to this article titled “An OS is an OS is an OS”.  The author argues that VMware ESX Server is an operating system; therefore, their (VMware’s) claims about bringing about the end of the OS is a bit hypocritical.  My initial (admittedly knee-jerk) reaction was to exclaim, “No it’s not!” and write something to that effect here.

Fortunately, I subdued the knee-jerk reaction, did some more thinking and reading, and have finally arrived at a point where I can see the author’s point and agree with him—to a certain extent.

The strict definition of an “operating system” has dramatically changed over the years, and from that perspective VMware does have a right to distinguish ESX Server from the typical OS.  The critical question here is, “What exactly comprises an operating system?”  This Wikipedia article (to which the original article’s author linked) states that:

“Operating Systems themselves have no user interfaces; the user of an OS is an application, not a person.”

The heart of ESX Server is the vmkernel, which has no user interface and can’t be observed or interacted with. By the strict definition of an OS according to Wikipedia, then you’re right—the vmkernel is an operating system, and VMware provides the Service Console as the “application” by which users can interact with the vmkernel’s managed resources.  In this case, then the original author’s complaint against VMware is valid—you can’t say that you’re going to eliminate operating systems by pushing your own operating system.  An OS is an OS is an OS, even if those OSes are very different in design and purpose.

Over time, though, the perception of an “operating system” has grown to mean more than perhaps the strict Wikipedia definition really entails.  If an operating system has no user interface, then how do we classify Microsoft Windows?  Or Mac OS X?  In this strict context, Windows would be defined as an application that runs on top of “something else” (the Mach kernel and kernel-mode subsystems, I suppose; it’s been ages since I took a hard look at the Windows internals) to provide access to managed resources. Back in the Windows 3.1 days, Microsoft got seriously torqued when people referred to Windows as “an MS-DOS application,” but that’s an accurate definition in the strict technical sense of the term.  Today, I’m sure that calling Windows “an application” would get Microsoft equally torqued.  (Note that Linux is not so plagued by this problem, since the X Window System is clearly an application that runs on top of the OS, as is the text-mode shell.)

Further complicating the matter is the announcement of ESX Server  3i, a slimmed down version of the hypervisor that can be embedded in server firmware.  Is the argument still the same?  Is the vmkernel still an OS then?  If so, what is “the application” by which users will interact with the vmkernel?  If we have broadened the definition of an OS to include the operating environment in which a user works (this would be the GUI in Windows, or the Aqua layer in Mac OS X), then the vmkernel no longer has that component.  It has no significant user interface.  Is it still an operating system?

<aside>It’s true that persistent beta users have discovered the presence of a command-line interface on ESX Server 3i, but it’s unclear at this time how pervasive the support for that CLI truly is.</aside>

I suppose this all sounds like a pretty trivial discussion, but at a deeper level this goes back to some thoughts I had (along with others) about the future of the OS.  In that article, I stated (sorry I’m quoting myself here):

If you’re an operating system guy, you’ll say that the OS has a bright future, and point to developments such as built-in paravirtualization and bundled hypervisors to prove your point.  If you’re a virtualization guy, you’ll say that the OS is dead, and you’ll point to developments such as third-party paravirtualization and independent hypervisors to prove your point.

See, Microsoft has to make the claim that ESX Server is an operating system, so that it can level the playing field—at least with regards to perception—between VMware’s virtualization solution and Microsoft’s own virtualization solution.  After all, if you’re VMware and you’re saying that your solution runs “on the bare metal, beneath the OS,” then this sounds somehow better than Microsoft’s hypervisor, which is “bundled with the OS.”  See the difference?  If Microsoft allows the perception that ESX Server is not an OS to grow, then it undermines their solution.

Likewise, VMware has to distinguish their solution from the “traditional OS” because if they don’t, then they begin to lose the perception advantage against other virtualization solutions such as Xen or Windows Server Virtualization.  You can see this trend even now on the VMware web site, where you’ll find statements like this about ESX Server 3i:

ESX Server 3i is the only hypervisor that does not incorporate or rely on a general-purpose operating system (OS), eliminating many common reliability issues and security vulnerabilities.

Is ESX Server an “operating system”?  What about ESX Server 3i?  As with so many other things in life, I guess that truly depends upon your perspective.

A new version (B52, build 3812) of Quicksilver was released in the last few days, and today I upgraded to the new version.  As I have stated here more than a few times, Quicksilver is one of the most useful—if not the most useful—utility I have installed on my MacBook Pro.  Yes, it’s that good.

In any case, I found that after upgrading Quicksilver, I couldn’t hide the Dock icon.  I know that I shouldn’t have worried about it, but it bothered me.  I want Quicksilver to run, but I don’t want the Dock icon there.  After much searching and testing, I had almost resigned myself to having the Quicksilver icon in the Dock when I finally came across an posting on the Blacktree forums that gave me an idea.  I had seen this post before, but had dismissed it as not useful to me; this time, however, I took another look.

It appears that the ability to hide Quicksilver’s Dock icon requires read-write permissions to the Quicksilver application itself.  Since I run as a non-admin user for day-to-day operations, I did not have read-write permission to the /Applications folder where Quicksilver was located and therefore did not have the ability to hide the Dock icon.  Supposedly, you should be able to edit the Info.plist file (found in /Applications/Quicksilver/Contents) and change the LSUIElement value to 0, but I tried this and it had no effect (no effect, that is, other than to change the value of the “Show the Dock icon” preference for Quicksilver but leave it greyed out and disabled).

After a series of other attempts, I finally granted myself read-write access to the application itself.  Voila!  That fixed it.  I was now able to show and hide the Dock icon at will.  What was really interesting, though, was that even after setting the application to hide the Dock icon, removing the read-write permission caused the Dock icon to reappear.  Very strange!

So, if you upgrade Quicksilver and find that you can no longer hide the Dock icon, grant yourself read-write permissions to the application and that should fix the problem.

UPDATE:  I find that the Quicksilver Dock icon still appears when I log in, requiring a quick series of checking the box, restarting, unchecking the box, and restarting again.  I still have no idea why this is the case.  Anyone have any suggestions?

UPDATE 2:  An update to Quicksilver (Beta 53, Build 3814) fixes the issue, as far as I can tell.

Have you ever been bitten by the apparent disconnect between the various groups involved in a technical implementation for a customer?  A colleague and good friend of mine (thanks, Tim!) shared this with me yesterday, and I found it quite humorous and—unfortunately—all too accurate for many organizations.  Have a look; I suspect you’ll find it rather amusing as well:

http://dotnet.org.za/photos/ernst/images/45781/original.aspx

How true it is…

In the event you need to disable clipboard redirection and can’t (or don’t want to) use Group Policy within Active Directory, there are settings you can add to the default .RDP file for the Remote Desktop Connection client software that will prevent clipboard redirection.

In my particular case, I was working on a VDI project and we wanted to be sure that the connection broker (Leostream) disabled clipboard redirection for all hosted desktops.  The broker already supplied some basic settings for the RDP connections, but disabling clipboard redirection wasn’t there.  Fortunately, I found a site that had a listing of RDP file parameters (not from Microsoft, surprisingly!).  The listing isn’t annotated with explanations of each setting, but they are—for the most part—pretty self-explanatory.

To disable clipboard redirection, add the following to the .RDP file:

redirectclipboard:i:0

For the ActiveX RDP control, you would use this text:

MsRdpClient.AdvancedSettings2.RedirectClipboard = FALSE

I guessed on the ActiveX control configuration, but it worked.  I haven’t found that information documented anywhere just yet.

There are also Registry changes that can be made on the client to disable clipboard redirection (see here).  And, as I mentioned at the beginning of this post, you can also use Group Policy.

I guess I’m on a bit of a NetApp kick this week.  After discussing (or perhaps revisiting) the idea of recovering files inside VMs using NetApp Snapshots (first here late last year, then again here), I wanted to take a closer look at full VM recovery using NetApp Snapshots.

First of all, it should go without saying that you should never use any of the procedures I’m describing here without first testing them yourself.  While they worked fine for me, they may not work fine for you.  Don’t just assume they will!  Do the due diligence and test it in your environment first; you’ll be glad you did.

Second, before using NetApp Snapshots to recover VM data (file-level or full VM), be sure you are getting good, consistent Snapshots.  The Network Appliance Technical Reports Library has a number of excellent articles on this subject; I’ll defer you there for more information.

I’ll break this article into two sections, one for block-level storage (I’m using iSCSI, but the process should be almost identical for Fibre Channel) and one for NAS/NFS.  Please note that I’m not focusing so much on the specific steps that are required as I am on general concepts and any gotchas that may arise during the process.

Full VM Recovery using Block Storage

To recover a full VM using block-level storage, a number of steps have to be taken:

1. Create a LUN clone (or a FlexClone) of the original LUN based on a Snapshot. 
2. Enable resignaturing on the ESX host(s) that will need to see the cloned LUN.
3. Mount the cloned LUN(s) on the ESX host(s) and copy the appropriate VM files from the clone to the production LUN.

For the first two steps, I’ll refer you back to one of my first articles on VMware data recovery with Snapshots, which has more information on the necessary commands and settings.

For the third step, you’ll need to login to the Service Console (typically via SSH) and copy the desired VM(s)—and all their files—from the cloned datastore to the production datastore, overwriting whatever is in the destination (you typically wouldn’t need to recover a full VM unless the production VM was hosed, right?).  Once the file(s) have been copied back over to the production datastore, dismount the cloned datastore and destroy it.

You should now be able to boot up your VM at the state it was in at the time of the Snapshot used to recover it.  Unless the Snapshot was a cold Snapshot (taken while the VM was powered off), the VM will perform a file system check (chkdsk or fsck) when it boots up.

Full VM Recovery using NFS

The procedure for recovering full VMs when using NFS is even easier:

1. Using an NFS client, mount the NFS export and navigate to the hidden “.snapshot” directory.
2. In the “.snapshot” directory, find the Snapshot from which you wish to recover the VM.
3. Copy that VM’s files (the entire folder) out of the “.snapshot” directory into the production filesystem, replacing the current contents (again, this assumes that what’s in the production filesystem is no good, else why would you be recovering a full VM?).
4. Unmount the NFS export from your NFS client.

The recovered VM should now boot and be back to the point in time at which the Snapshot was taken.  Again, unless the Snapshot was a cold Snapshot, the VM will likely perform a file system check upon boot.  This is normal and not unexpected.

I suppose you could even do this second procedure from a CIFS client, assuming that CIFS and NFS were both configured on the storage system and an appropriate CIFS share existed.  (Please note that I’ve never tried this, so I can’t tell you what the results might be.)  In that case, use the “~snapshot” directory instead of “.snapshot”.

And that’s it—there you have two ways of recovering entire VMs using Network Appliance Snapshots.  As always, feel free to hit me up in the comments with any questions, thoughts, corrections, or rants (just keep the rants on-topic, please!).  Thanks for reading!