blog.scottlowe.org

I’ve targeted Solaris (specifically, Solaris 10 on x86) as the next major technology that I’m going to try to learn.  I’ve always been fascinated with UNIX and UNIX-like operating systems such as Linux, and Linux’s popularity on the x86 platform made it much easier to learn because I didn’t have to acquire any exotic hardware.  With Sun’s (apparent) renewed interest in x86/x64, Solaris is much more accessible now than it was in the past.

Obviously, I’m not a complete newbie to the Solaris environment, having written a couple articles on Solaris-AD integration (the latest being found here).  However, I don’t feel like I have a solid understanding of the operating system and its architecture, and I’d feel much more comfortable with that information under my belt.  At some point, the IT industry being what it is, I’ll need to seek some sort of Solaris certification, but that’s not my primary goal.  Understanding the product itself is my primary goal; certification will merely be a side effect.

Here’s what I’ve done so far:

• I’ve created a Solaris 10 (32 bit) virtual machine on my VMware ESX Server farm; that’s been the system I’ve used mostly for testing the Active Directory integration instructions.  I’ve also done some work with the automounter (automounting home directories via NFS).
• I’ve also just recently gotten a Solaris 10 (64 bit) VM running under VMware Fusion on my MacBook Pro; I’ll use that system for more day-to-day operational tasks and getting used to the interface.
• I have a group of Solaris- and UNIX-related RSS feeds in NetNewsWire, including BigAdmin (What’s New and Feature Articles), Inchoate Curmudgeon, and a del.icio.us tag feed, among others.

I’d certainly appreciate any suggestions from those who may have already been down this path as to specific projects I should undertake, books I should acquire, websites to frequent, RSS feeds to which I should subscribe, etc.  In addition, any guidance as to how I should balance Solaris vs. OpenSolaris (on which one should I focus more effort?) would be very helpful.  And what builds of Solaris/Solaris Express are most beneficial to use?  I’m currently using Solaris 10 Update 3, but I’m not sure if a different build would be better to work with.  That’s the kind of information that would be great to get from those wiser and more experienced.

Wish me luck!

In the short period of time since I mentioned the Virtual Machine MKS Client in my post on assorted VMware tools, the client has gone from version 1.1 to version 1.8 and has added tons of functionality.  Some of the features include:

• The ability to logon to either VirtualCenter or the ESX Server directly
• Multiple view options, including full screen and borderless
• Ability to send Ctrl-Alt-Del signal to VM
• Warning/alarm feedback from VirtualCenter into the MKS client
• Full power off/suspend/power on/reset functionality

I’m sure there’s more, but that should definitely get you started.  It looks like an absolutely fantastic application, but unfortunately I can’t use it because it’s designed to run on Windows and I primarily use a Mac.  Any enterprising MKS programmers out there (per Mike Laverick, apparently there’s quite the MKS programming contest going on) want to create an MKS client for Mac OS X?

I just saw this headline from virtualization.info about Cisco being the first to announce a third-party virtual switch for ESX Server. Honestly, I’m not too terribly surprised.

Surely, the announcement of John Chambers as a keynote speaker at VMworld 2007, followed by the investment of $150 million by Cisco into VMware surely had to signal that something was going on.  This revelation that Cisco plans on being the first to announce a third-party virtual switch for ESX Server is honestly not a surprise.  In fact, if I recall correctly, there were rumors of this last year at VMworld.

Still, despite the fact that it’s not really a surprise, I’m still excited about the possibility that this is actually the case.  I’m not a Cisco expert, but I’d love to have the ability to run Cisco-based switches on ESX Server (assuming the overhead isn’t too significant).  That should make integration of ESX Server with Cisco-based networks much easier than it has been in the past.  What’s next—SAN switches on ESX?  (I may not be too far off, there, with all the rumors and discussions about NPIV support in ESX Server.)

UPDATE:  Ryan Glover over at p2vd.com shares his thoughts on the announcement here.

How many of us would like to be indestructible?  To know that no matter what happens, no matter what comes against us, we cannot be destroyed or killed?  It would be pretty cool, right?

Well, here’s a newsflash, given to me courtesy of Pastor Greg Laurie on the radio yesterday on my way to Charlotte, NC for a meeting:  for Christian believers, we are indestructible.

That’s right, we’re indestructible.  To paraphrase what Pastor Greg had to say:  Until God is ready for us to come home to Heaven, we are indestructible.  No plots schemed against us, no attacks targeted at us, no lies told about us—nothing—can destroy us.  Why?  Because it’s not God’s will!  When God is ready for us to leave here and go to Heaven (i.e., when our time is up), we’ll go.  Until that time, we won’t.  It’s as simple as that.  Until God is ready for us to be with Him in Heaven, we are indestructible.

Of course, that doesn’t mean we should test God, and Pastor Greg pointed that out.  But what it does mean is that we don’t have to live in constant fear.  Should we take reasonable steps to protect ourselves, our families, and our friends?  Of course.  God gave us a brain so that we would use it.  But after taking reasonable steps, we don’t have to live in fear.  Why?  You’ve got it—because we’re indestructible!

So, if you’re facing a challenge, facing a mountain ahead of you, if your opponents are lining up against you (and believe me, I’ve been there before), take heart and be encouraged.  You, my Christian friend and believer, are indestructible.

So there’s been a flurry of coverage over the last couple of weeks regarding a statement from Microsoft regarding WSV’s lack of live migration (aka VMotion) functionality.  As I mentioned when I discussed the announcement of features being cut from Windows Server Virtualization (WSV, or “Viridian”), the lack of live migration functionality really is, in my opinion, a serious stumbling block for Microsoft.  I wasn’t the only one that thought so, either.  (There was a post about it on the VMTN Blog at some point as well, but I can’t find a link for it now.)

After Microsoft announced that live migration wasn’t going to make it into WSV and virtualization-savvy users responded with numerous statements about how this killed WSV’s chances against VI3, Microsoft apparently realized that live migration was, in fact, important to customers.  (Apparently customers don’t like downtime.  Who knew?)

To help alleviate the bad press over the lack of live migration in a product that is already pretty far behind the competition, Microsoft now started spewing stuff about quick migration:

“The recent press has been inaccurate to say we don’t do migration - we do migration: quick migration,” Lees said Wednesday. Live migration is a memory-to-memory system while quick migration is machine-to-machine and disc-to-disc.

Mr. Andy Lees (speaking in the quote above taken from this article from The Register) goes on further to say:

Lees called the debate between quick migration and live migration a “red herring” based on six seconds of difference, which mattered only to disaster recovery. On that basis, Windows Server 2008’s planned geo-clustering feature would help users out of any squeeze and could - he claimed - beat VMware.

Yes, but it’s still downtime, Mr. Lees!  That’s the point—without live migration, any migration solution still involves downtime, and therefore can’t really compete with VMotion.  Host-level clustering is great, and something I’d love to see VMware tackle for those shops where VMware HA just isn’t quite enough to meet SLAs.  But host-level clustering is no substitution for VMotion.

Even this blog posting (from an MS employee, no less) admits that Quick Migration is really nothing more than Host Clustering.  This isn’t new technology; this is just Microsoft repackaging the same old stuff again.  (Unless memory serves me incorrectly, Microsoft has been talking about clustering hosts for a while now.)

Microsoft can change the names of features to make them look more like their competition, but the fact of the matter remains:  when Windows Server Virtualization debuts in late 2008 (supposedly within 180 days of the release of Windows Server 2008, which is due to be released in late February 2008), it still won’t have a feature set comparable to the feature set that VMware has today.

The idea behind an Active Directory “delayed replication DC” (also referred to as a “slow DC” or a “lag DC”) is that organizations can more quickly recover portions of their Active Directory structure by performing an authoritative restore from this delayed replication DC instead of having to go back to tape.  Keep in mind that many organizations use off-site storage of backup tapes, and recalling backup tapes from the off-site storage facility can take some time, as can the tape operations themselves.  Let’s face it: when it comes to speed, tape isn’t exactly king of the hill.

With a delayed replication DC (this would be a DC that only replicates on specific days of the week or after an extended period of time, like 72 hours), however, that process can be shortened drastically because recovering an OU, a user object, a group, or even AD-integrated DNS data can be done directly from the delayed replication DC.

First, let’s look at setting up a delayed replication DC, then have a look at the process of performing an authoritative restore from that DC.

Setting Up a Delayed Replication DC

Setting up a delayed replication DC (hereafter referred to as a delayed-rep DC) is not terribly complicated:

1. Create a subnet object with a 32-bit mask representing the IP address of the new DC that will become the delayed-rep DC.  For example, if the new DC will have the IP address 192.168.213.150, create a subnet object (in Active Directory Sites and Services) for 192.168.213.150/32.  Doing it this way means that we don’t have to create a new subnet or VLAN and potentially waste IP addresses, and simplifies the networking configuration (no need to worry about routing tables or anything like that).
2. Create a new Active Directory site and link the new subnet you just created to that site.
3. Create a new site link between the new AD site and an existing AD site with production domain controllers.  Don’t worry about setting the replication interval just yet; we’ll do that after the initial replication takes place.
4. Create a new GPO, linked to this new site, that will prevent registration of specific DNS SRV records for the delayed-rep DC.  This is the only (relatively) complicated part.  This article by Gary Olsen (a well-respected AD expert) provides more information on the specific settings that should be included in this GPO.  (More information available here and in Microsoft KB306602 as well—refer to the “To Configure Domain Controllers or global catalogs to Not Register Generic Records” section for Windows Server 2003.)  When specifying the mnemonics to prevent the registration of DNS SRV records, be sure to allow the A record (for the hostname-to-address lookup of the server) and the GUID CNAME record (used for establishing replication connections).
5. Promote the new DC (making sure its IP address matches the address specified during the creation of the subnet above!) and it will automatically place itself into the delayed-rep site.
6. Once you are satisfied that initial replication is complete, adjust the replication schedule on the site link to the delayed replication DC.

See, that wasn’t so bad!  The cool thing is that you can use a virtual machine as your delayed-rep DC to further save some costs/rack space/equipment.

Once the delayed-rep DC is up and running, then you are ready to use it to perform authoritative restores of AD objects.

Performing Authoritative Restores from the Delayed-Rep DC

The process for performing an authoritative restore using this delayed-rep DC is also pretty straightforward.  Again, there is one minor complication:

1. Reboot the delayed-rep DC into Directory Services Restore Mode (DSRM).  You’ll need the DSRM password to log in; you do know it, right?  (If not, you can reboot the DC normally, then use Ntdsutil to set the DSRM password; see here.)
2. Once you log on, use Ntdsutil to mark the desired objects as authoritatively restored (more on that in a moment).
3. Reboot the server into normal operation.
4. Force outbound replication from the delayed-rep DC to the rest of the organization.

It’s item #2 that’s the complicated part this time.  First, because it’s Ntdsutil, and not many people know/use Ntdsutil.  Second, because you must know the full DN of the object or objects you want to mark as authoritatively restored.

Fortunately, you can use ADSI Edit to help you find the full DN.  (Those of you that did not install the Support Tools on your delayed-rep DCs may take a moment to go and install them.  Done now?  Good, let’s continue.)  There are a couple of gotchas, though, even with using ADSI Edit:

• You’ll need to put double quotes around the DN if there are any spaces anywhere along the way. ADSI Edit doesn’t do this, so be sure to add them yourself.
• Any commas in the individual components of the DN must be escaped with a backslash (\) character.  For example, in situations where the user object is named “Smith, Bob” you’ll need to reference that as “Smith\, Bob” in the DN.
• Don’t forget to use the right components in the DN—CN=, OU=, or DC=.  More on that in a moment.

Let’s look at some examples.  First, let’s say we need to restore a user object named “John Doe” in the Raleigh OU under the Locations OU in the domain example.com.  The DN would look like this:

“cn=John Doe,ou=Raleigh,ou=Locations,dc=example,dc=com”

What if the user accounts are listed with lastname then firstname, separated by a comma?

“cn=Doe\, John,ou=Raleigh,ou=Locations,dc=example,dc=com”

What about a group?

cn=DnsAdmins,cn=Users,dc=example,dc=com

Note that we omit the double quotes here because there are no spaces in the DN.

What if what we want to restore isn’t a user object at all, but instead is an Active Directory-integrated DNS zone?  Good question!

In this case, the answer is “It depends”.  Upon what does it depend?  The zone’s replication scope:

• For zones with a replication scope of “All domain controllers in the Active Directory domain” (or if you are still running Windows 2000), then the correct location for these zones is “cn=MicrosoftDNS,cn=System,dc=example,dc=com”.
• For zones with a replication scope of “All DNS servers in the Active Directory domain”, then the correct location for these zones is “cn=MicrosoftDNS,dc=DomainDnsZones,dc=example,dc=com”.
• For zones with a replication scope of “All DNS servers in the Active Directory forest”, then the correct location for these zones is “cn=MicrosoftDNS,dc=ForestDnsZones,dc=example,dc=com”.

The tricky part about restoring AD-integrated DNS zones is the naming in the DN.  Here’s an example:

“dc=testlab.example.com,cn=MicrosoftDNS,dc=DomainDnsZones,dc=example,dc=com”

This would be for the zone testlab.example.com, configured as a AD-integrated zone with a replication scope of all DNS servers in the domain, in the AD domain example.com.  Key thing to note here:  pay attention to the use of “dc=” at the beginning of the DN.  Zones use “dc=” and not “cn=”!  (Fortunately this is correctly represented with ADSI Edit.)

OK, enough about DNs.  Once you have the DN and understand how it should be represented when using Ntdsutil, you can proceed with the actual authoritative restore itself.  Refer to this Microsoft web page for more details on the specific Ntdsutil commands to perform an authoritative restore.  Once the authoritative restore process is complete, reboot the delayed-rep DC normally and force replication with the rest of the organization.  Your objects should magically re-appear in Active Directory!

There’s more to this process than I’ve described here, but there are a ton of resources available to provide more in-depth information and explanations of the various scenarios.  This information should at least get you started.

I don’t like advertisements in my RSS feeds.  I just don’t.  It’s not that I begrudge the authors the ability to monetize their content; that’s their choice, and I can certainly understand the need to pay for hosting and bandwidth costs.  The day might even come one day when I am faced with the same issues here on this site.  Even so, I don’t like ads in the feeds.  After all, if it’s a good site, I am very likely to visit the site anyway, even with full feeds, so that I can comment, view others’ comments, or see related posts.

So, when ads started showing up in my RSS feeds from a variety of sources, I looked for a way to kill them.  I came across this article (at least, I believe it was this one—I’m not entirely sure) that suggested the use of userContent.css to block ads in NetNewsWire.  It works like a champ!

Basically, you have to edit the NetNewsWire stylesheet to include a reference to your ad-blocking code, like this:

@import url(../userContent.css);

Then put the userContent.css file of your choice (apparently there are many; I think I pulled down one of the files linked to in that article) in the ~/Library/Application Support/NetNewsWire/Stylesheets folder, restart NNW, and away you go!  No more pesky ads in the RSS feeds.

In addition to ads, the “FeedFlare” functionality offered by FeedBurner irks me as well.  If I want to digg it, bookmark it with del.icio.us, or whatever, I’ll do that—I don’t need links in the content of the feed to help me.  I suppose that functionality is useful to some readers, and probably helps increase readership and visibility of the site, but I still don’t like it.  Fortunately, that is easily blocked using this same technique.  You only need to tweak the userContent.css to include the additional URLs used by FeedBurner to add the FeedFlare content.  (By the way, this is not a knock against FeedBurner; I use FeedBurner too.  I just don’t like FeedFlare.)

Over the past few weeks, a number of VMware-related tools have been released.  All of these tools are third-party tools written by avid VMware fans or ISVs, and as far as I am aware all of these tools are available at no cost.

• First up is VMX Extras, a tool for editing the VMX files used by VMware Fusion (VMware’s Mac OS X product).  While you certainly can edit these files with a text editor (and I’m sure many of you have in the past), this makes it a bit easier.  In addition, there are some preconfigured options available as well for stuff like a BIOS delay (to make accessing the BIOS of a VM easier), enabling paravirt-ops (for guests that support it, currently only Ubuntu 7.0.4 and derivatives), and some others.
• Next is VCplus, a tool released by Richard Garsthagen of run-virtual.com.  Richard has a number of tools available on his site, including VMotion Info (for helping to determine CPU compatibility for VMotion), VM Time (for checking for time differences between guest and host), and the Virtual MAC Tool (for assigning fixed MAC addresses to ESX/VirtualCenter guests).  VCplus extends the functionality of VirtualCenter to include information such as disk usage inside the VM and if a VM has a snapshot (and if so, the size of the snapshot disk), and allows you to sync the DNS name of the VM to the display name.  More functionality is planned, so this is one to keep your eyes on.
• Russian company Veeam has been pushing out the VMware-related utilities, and the latest is one called EsxDiag.  I was unable to find a link to the product on Veeam’s website; I was alerted to EsxDiag via virtualization.info and there is a link there to the application download.
• Eric Sloof has released a beta version of his VMware MKS Client, which is a remote console application for VMs.  I haven’t tried this one yet, but it sounds like it might be handy.

I’d love to hear feedback from anyone actually using some of these tools already in their environments.  I have a few customers who could really use some of the functionality offered by some of these tools, but wouldn’t want to recommend them until I get some feedback on real-world usage.

I’m not really sure why, but for some reason I got on a kick to start using IRC.  So, for the last week or so, I’ve been regularly logging in to irc.freenode.net using Colloquy (a really great Mac OS X IRC client, by the way) and hanging around in the #vmware channel, helping people with their VMware-related problems and questions—and learning a little bit in the process.

Earlier this week I had someone who was trying to export a VM from ESX Server to VMware Workstation 6.0 using VMware Converter and kept getting an “guest OS not recognized” error.  At first, we thought it might be a problem with a missing BOOT.INI, an incorrectly configured BOOT.INI, or a missing MSDOS.SYS file (yes, even newer versions of Windows have that file, although it is an empty zero-byte file); any of these problems can cause that particular error message.  As it turns out, it wasn’t any of these issues.  The problem was running Converter on Windows Vista; as soon as the user loaded Converter on a Windows XP Professional system, it worked just fine.  Go figure.

In another instance, there was a user who wanted to install a custom-compiled Linux kernel in the Service Console of his ESX Server.  While I couldn’t find any hard and fast rules from VMware on that, I strongly encouraged him to avoid that path.  I suspect going down that road would have rendered his ESX Server unusable.

Finally, just earlier tonight I helped someone get a physical installation of Microsoft Windows running as a virtual machine under VMware Server on Ubuntu Linux.  That was pretty cool…I’d never done that before.  It turns out that this won’t work with default permissions in Ubuntu; you have to either run as root (which is disabled by default in Ubuntu) or add your user account to the “disk” group.  This is the only way to have permission to the /dev/hd* device that represents the physical installation of Windows.

Any other #vmware users out there?  I’d love to hear some of your stories from others you’ve helped and lessons you’ve learned.

Although much of the administration of servers running VMware ESX Server 3.0 will occur in the Windows-based Virtual Infrastructure client connected to a VirtualCenter server, there are times when it is quicker or easier to perform an administrative task directly on the ESX Server itself—either via the command-line interface (CLI) or via the VI client authenticating directly against the ESX Server.  The problem with this is that, by default, administrators will have to use different credentials when connecting the VI client to ESX Server directly.  In addition, these credentials must be managed separately from Active Directory, and separately on each individual ESX Server.  As the number of ESX Servers in a farm grows, this can quickly become an administrative nightmare.

Fortunately, we can fairly easily integrate ESX authentication into Active Directory, so that the same account used in VirtualCenter is also used when logging in to the ESX Server directly.  While VMware took the step of automating a portion of this process for us in the esxcfg-auth command, it only takes us part of the way.

Let’s take a look at what part esxcfg-auth does accomplish for us, then look at how to accomplish the rest of the task.

Using esxcfg-auth

The esxcfg-auth command will help automate a good portion of the work required to integrate ESX Server into Active Directory; specifically, it will automate the Kerberos configuration.  To use esxcfg-auth to enable AD authentication, use the following command (lines are wrapped for readability; the backslash indicates line continuation):

esxcfg-auth --enablead --addomain=example.com 
–addc=dc1.example.com

Obviously, you’ll want to substitute the appropriate values for “example.com” and “dc1.example.com” on the command above.  So what does this command do, exactly?  Here’s the breakdown:

• Modifies the /etc/krb5.conf file to use example.com as the default Kerberos realm, and to use dc1.example.com as a KDC for that realm.  In this same file, the domain “example.com” is mapped to the realm “EXAMPLE.COM” (keep in mind Kerberos realms are always specified in UPPERCASE).
• The /etc/pam.d/system-auth file is modified to use pam_krb5.so for Kerberos authentication.

What does this command not do?  Well, for one it doesn’t configure the ESX Server for anything other than pure authentication.  This means that although users will be forced to authenticate against Active Directory, ESX Server still expects to find the accounts defined in the local /etc/passwd file.  This means that password management is centralized, but account management is still decentralized.  (Some might see this as a security advantage, in that we must manually define an account in order to allow that account to log in to that server.)

To fully centralize account management, we’ll need to step outside of the esxcfg-auth framework and get our hands dirty.  Ready?

Finishing esxcfg-auth’s Work

To fully round out the authentication/account management configuration, Active Directory will have to be made aware of some UNIX-specific attributes.  This means extending the schema.  If you are running Windows 2000 or Windows Server 2003 pre-R2, this means installing Services for UNIX (SfU) 3.5; for Windows Server 2003 R2 or later, this means installing Identity Management for UNIX.

I’ll refer you to this article for Windows 2003 and Windows Server 2003 pre-R2 and this article for Windows Server 2003 R2 or later for more information.  (Because the ESX Server service console is based on Red Hat Enterprise Linux, these Linux-AD integration guides are very applicable here.)

Once Active Directory has been configured, the only tasks left for us to do are 1) to configure the nss_ldap libraries; 2) to configure /etc/nsswitch.conf to enable LDAP for naming services; and 3) verify time synchronization.

To configure the nss_ldap libraries, you’ll need to modify the /etc/ldap.conf file as described in Step 5 of the “Prepare Each Linux Server” section of this article (assuming you are using Windows Server 2003 R2).  This sets up the connection to Active Directory via LDAP and configures the attribute maps accordingly.

Please note that you’ll need to create an account in Active Directory (a Domain Guest is fine) that the nss_ldap libraries may use for LDAP queries.  You’ll specify that account information when configuring /etc/ldap.conf.

Next, you’ll need to configure /etc/nsswitch.conf.  You only need to modify the passwd, group, and shadow lines, and you only need to add “ldap” at the end of the lines.  The lines will end up looking something like this:

group:    	files ldap
passwd:    	files ldap
shadow:    	files ldap

At this point, you should be able to test the nss_ldap configuration.  Run “getent passwd <AD user account>” and you should get back information about that account’s home directory, login shell, UID, and name.  If you don’t get back any information, go back and double-check your configuration.

To verify time synchronization, have a look at the NTP configuration found in /etc/ntp.conf.  Make the changes you need here to be sure that both Active Directory (the forest root PDC emulator, specifically) and ESX Server are both synchronizing time and will stay in sync.  Otherwise, the Kerberos authentication process will fail.  Keep in mind that you may need to adjust the Service Console firewall using esxcfg-firewall in order to allow the appropriate traffic outbound.  (Thanks to KentA for reminding me about this step!)

Once “getent passwd” is working as expected and time is in sync, then you should be able to SSH into the ESX Server with any appropriately configured AD account (i.e., any AD account that has the “UNIX Attributes” tab completed with valid information).  This gives you a similar level of control over who is allowed to login and who isn’t; accounts that don’t have any UNIX attributes won’t be able to authenticate and gain access to the ESX Servers.

In addition, you should be able to configure some level of access control as described here.

Summary

To summarize, the process for integrating ESX Server into Active Directory looks like this:

1. Use esxcfg-auth to set up the Kerberos authentication.
2. Extend the AD schema (if necessary) to include UNIX attributes such as login shell, UNIX home directory, UID, UID number, etc.  This is accomplished in different ways depending upon the version of Windows in use.
3. Populate the UNIX attributes for those user accounts that should be allowed to access the ESX Server(s).
4. Configure /etc/ldap.conf on the ESX Server to configure LDAP connectivity back to Active Directory for name service lookups.
5. Configure /etc/nsswitch.conf to use LDAP for name service lookups.
6. Verify (and configure, if needed) time synchronization via NTP on the ESX Server.
7. Test the configuration using “getent passwd” or “getent group” or both.

This configuration will centralize not only authentication for the ESX Servers but will also centralize account management in Active Directory as well.

Please feel free to add any corrections or suggestions for improvement in the comments below.  Thanks!