blog.scottlowe.org

Reader Scott Merrill pointed out something to me in an e-mail regarding a Registry change that might be necessary in some Active Directory integration scenarios:

Finally, I would like to share one registry change that we’ve found to be necessary in our AD integration. By default, the MS LDAP server only returns 1,000 results. As a university department with more than 1000 active students, this limitation has caused us some frustration.

This KB article shows how to increase the number of results returned in a query: http://support.microsoft.com/kb/315071

We recently set MaxPageSize to 5,000. I don’t know if this will
introduce additional problems down the road, but at least it lets me fully enumerate all our AD users from a Linux machine with `getent passwd`.

If you have an Active Directory domain with more than 1,000 users in the DN specified in your LDAP configuration, then this is a Registry change you’ll want to investigate. Otherwise, you could find that your UNIX/Linux servers aren’t able to fully enumerate all the users in the domain.

Thanks, Scott!

Reader Jeffrey Spear contacted me a while back with some problems he was experiencing in trying to integrate some Linux systems into Active Directory. Basically, Kerberos was working but LDAP wasn’t. He was able to use “kinit <AD username>” to generate a Kerberos ticket, but using the “getent passwd <AD username>” was not working. No error messages, nothing; it just didn’t work.

We traded e-mails back and forth for a while, and eventually he found the solution himself:

We work with a locked down version of OSs and in this case a domain policy on the Windows server was preventing the RHEL machines from accessing account info.  The policy was “Domain controller: LDAP server signing requirements” which was set to “Require signature.”  When I changed this setting to “None” it worked great.

This is good information and important to keep in mind; I’ll be sure to incorporate this into the next revision of the Linux-AD integration instructions. (No, I don’t have a timeframe on when that will be!)

In the meantime, if anyone has a workaround for this problem that will allow LDAP to work with signatures enabled or required, I’d love to hear it. Speak up in the comments below!

A customer with whom I had worked to fully integrate their ESX Server systems into Active Directory—using my instructions here—had run into a problem. The problem was that local logins were refused when the Service Console lost network connectivity. Clearly, this was a problem; if the customer couldn’t login as root when the network is down, even locally at the console, then we have problems. So I set out today to isolate and fix the problem.

After much trial and error, I had determined what was not the cause the problem:

• The /etc/pam.d/system-auth file was not at fault; we tried numerous combinations in system-auth and there was no difference in behavior
• The /etc/ldap.conf file was not at fault; we even tried adding a few additional entries (like “bind_policy soft”) to help with issues when LDAP was down and not responding
• A lack of DNS resolution was not the problem; the behavior was the same whether DNS was working or not

Finally, I was able to track down this thread which discusses the behavior of the nss_ldap libraries when the LDAP service is not available across the network. In this specific message in the thread, it is noted that nss_ldap will try to contact LDAP to enumerate group membership, even if LDAP is down. So the problem was with using LDAP for group membership, and a quick edit to /etc/nsswitch.conf to remove LDAP from the group line proved that to be true.

As shown in the message, the only workarounds are:

• Upgrade to v245 of nss_ldap, which allows the use of the “nss_initgroups_ignoreusers” directive; this instructs nss_ldap to not perform group enumeration for the specified users; or
• Remove the “ldap” entry from the group line in /etc/nsswitch.conf.

Unfortunately, ESX Server 3.0.2 and ESX Server 3.5.0 only supply nss_ldap v207-17, which is too early to support that directive.  Of course, we can’t really upgrade the library, either, since that’s not supported by VMware. So the only real fix for VMware ESX Server AD integration scenarios is to not use Active Directory for group memberships. User accounts can still be managed using Active Directory—and authentication occurs against Active Directory—but groups and group membership will have to be handled locally.

This issue is applicable to other operating systems besides ESX Server, though, and for those operating systems an upgrade of the nss_ldap library and the use of the “nss_initgroups_ignoreuser” directive in ldap.conf may be all that is needed to fix an issue with local logins being refused when network connectivity is not present.

UPDATE: It appears that local logins will work without network connectivity, even with full Active Directory integration, if you use the Emergency Console. Thanks to Magnus for the update!

I wanted to take just a brief moment and let everyone know about a few articles that I’ve got “in the pipeline” for the site.  If there is one—or more—of these articles that looks particularly interesting, speak up in the comments.

Here are the articles that are currently under development:

• An update to the Solaris-AD integration instructions:  Last time I ran through these instructions I came across a number of discrepancies and little “gotchas.”  I need to incorporate the workarounds into the integration instructions and publish a new version.
• A brief blurb on NetApp OSSV:  NetApp’s Open Systems SnapVault (OSSV) is a pretty cool technology, so I want to take a quick look at setting it up.  I’m also exploring to see what kind of unique synergies may arise from using OSSV in a VMware environment.
• Restricting access to ESX Server when using AD integration:  As reader Scott Garrett points out in this comment to an earlier article, ESX Server’s version of sshd doesn’t support the UsePAM directive.  This prevents us from using group membership in Active Directory to control access to ESX Server’s Service Console.  Or does it?  I have a hunch that there may be at least one workaround for this problem; once I’ve tested it, I’ll document it here.
• New iSCSI functionality in ESX Server 3.5:  VMware appears to have refreshed the software iSCSI initiator in ESX Server 3.5, so I want to take a closer look at and discuss here some of this new functionality.

That’s it for now, although I’m certainly open to other items that pique reader interest.  Feel free to submit your suggestions in the comments.  Thanks!

This afternoon, I walked back through my own instructions for integrating Solaris 10 and Active Directory, and I found that the process wasn’t as smooth as perhaps I’d believed it to be.  As a result of walking back through the process again myself, I’ve collected some notes.  At some point in the near future, these notes will be integrated into a new version of the Solaris-AD integration instructions.

So, without further ado, here are the notes I collected in no particular order:

• The Blastwave Samba package does not create it’s own smb.conf file in /opt/csw/etc/samba.  This is correctly pointed out in the latest integration instructions, but I wanted to mention it again here.  You’ll need to manually create the /opt/csw/etc/samba/smb.conf file before attempting to join the Solaris server to Active Directory via the ‘net ads join’ command.
• The defaultServerList portion of the ‘ldapclient manual’ command only supports IP addresses.  The LDAP client service kept going into maintenance mode when using hostnames.  On a hunch, I substituted IP addresses for hostnames, and it worked.  Go figure.
• Apparently, you can’t use ‘ldapclient mod’ to change an existing attribute map.  I had a hunch about resolving a co-existence issue where both Solaris and Linux are both authenticating against Active Directory—more on that particular topic is coming soon as well—and needed to change the attribute maps for the homedirectory and loginshell attributes.  I ended up editing the ldap_client_file manually (found in /var/ldap; must be made writable using chmod) in order to make the change.  If anyone has a more elegant fix, please let me know.
• The ‘net ads join’ command correctly creates a Kerberos keytab with the appropriate entries, but places it in the wrong location.  On my test system, it placed the krb5.keytab file in the /etc directory, and Solaris expected it to be in /etc/krb5 instead.  Until I moved that file, authentication against Active Directory consistently failed.
• It turns out that it’s not really necessary to enable the DNS client using ’svcadm enable svc:/network/dns/client:default’; from what I’ve been able to gather, that’s there as a dependency only.  The ‘nslookup’ and ‘host’ commands seemed to work just fine with this service still disabled.

Again, I’ll be incorporating these changes into a future version of the Solaris-AD integration instructions.  I hope to have that complete within the next week or two, so stay tuned.  In addition, I have information coming to help with the co-existence of multiple UNIX and UNIX-like operating systems all authenticating against the same Active Directory forest, so keep your eyes peeled for that as well.

A reader kindly shared with me some tips and tweaks he used to help resolve some performance issues with Linux-AD integration, and now I’d like to share them with you:

I ran into some nagging performance issues with Linux/AD integration the other day, and managed to solve them (mostly)—I thought you might be interested in the solution.

Since I’m not exactly following your integration guide (I use DNS lookups to locate LDAP servers in AD, and use GSSAPI authentication for nss_ldap instead of a binddn), I have a bit more overhead on my getent system calls, that was starting to get noticable when it came to getting directory listings, etc.

Two things I have done to alleviate this issue:

1. Since we have a flat catalog, and all of our DCs are also GCs, I
   disabled recursion. This is not universally relevant, but it does assist me
2. I edited /etc/nscd.conf to set a 5 minute cache time for passwd and group (but not host!) entries. Data is still usually piping fresh, but now we only call nss_ldap once every 5 minutes, instead of every time we need to know who owns a file. Then I started nscd, and set it to start on boot.

NSCD is a standard part of most RHEL and derivative installs.

Since implementing the above, the user-level experience is no longer
discernably different than the old /etc/passwd method of authentication/authorization.

This is good information to have.  Thanks to Brandon for sharing his experience!

I’ve received some feedback from a reader who alerted me to some sort of interaction between the Local Security Policy on the Windows side and Linux servers authenticating to Active Directory via Kerberos/LDAP/Samba.  I haven’t quite been able to get to the root issue yet, but here’s the high level overview.

The reader was seeing strange delays at the end of a Linux logon process that seemingly could not be explained.  After jumping through all the hoops, another administrator within the organization changed the Local Security Policy setting that governed the use of LM and NTLM authentication, and the delays disappeared.

The policy had been set to allow both LM and NTLM authentication; when changed to allow only NTLM authentication, the delays disappeared immediately.  The Linux server in question did have Samba installed, so apparently Samba was timing out trying the LM authentication; this caused the delays.  Of course, this is all just speculation, as we don’t know exactly why the policy change eliminated the delay.

In any case, since I’ve been pushing the use of Samba in my latest integration instructions (Solaris version here), I thought it might be prudent to mention this feedback.  In the event you start seeing some strange delays in your Linux authentication requests, check the Local Security Policy and see if LM authentication is being permitted.  That might just be your culprit.

I was going through my list of flagged headlines in NetNewsWire and realized that I’d built up quite a list of articles that I intended to write something about.  Some of them just don’t merit a full-blown post, though, so I thought I’d just toss a bunch of them in here along with a brief sentence or two about them:

• VMTN Discussion Forums: vdiskmanager GUI for OSX:  An enterprising Fusion user has written an OS X GUI for vdiskmanager, so that VMDKs on Fusion can be expanded or defragmented, or new virtual disks can be created.  I haven’t tried it yet, but it looks like it could be extremely useful, and it’s nice to see Fusion users creating useful utilities like this.
• Running ESX 3i Beta in a VM with VMware Fusion:  Still thinking Fusion, this article discusses how a user managed to get ESX Server 3i (the beta version obtained at VMworld 2007) running as a VM under Fusion.  There’s also information on running it under Workstation 6 as well.
• Tech: How to get the command line in ESX Server 3i beta:  Turns out ESX Server 3i has a command line after all, based on BusyBox.  Richard Garsthagen has more information about ESX 3i available at run-virtual.com.  Also see Eric Sloof’s info on boot options.
• Storm Worm Botnet Attacks Anti-Spam Firms:  Is this botnet really as massive as everyone says?  I’ve been seeing so many articles about the Storm botnet, but I have yet to see (perhaps I haven’t looked hard enough yet) definitive information that describes the type of traffic these bots generate.  Surely there’s got to be something we can do about this.
• Microsoft Updates Windows Without User Permission, Apologizes:  Oh, goodness—where do I start with this one?  Let’s just say that I’m glad I’m using Little Snitch, which catches this kind of outbound traffic that so easily slips through the Windows “firewalls” onto the Internet. Otherwise, I might be getting product updates without anyone bothering to tell me so.  (And perhaps it’s just me, but an apology from Microsoft doesn’t make me feel any more trusting of them.)
• NFS vs iSCSI vs FC:  More information on why we should be interested in running VMware over NFS.

I guess that’s all for now, as it’s getting late and I have to get up in the morning and go to church.  Feel free to share any comments or corrections below.  Thanks for reading!

Regular readers of this blog know that I like to work on integrating various systems into Active Directory.  I’ve written a couple of articles on the issue:

Linux-AD Integration, Version 4
Solaris 10-AD Integration, Version 3
Active Directory Integration Index

These articles have been pretty successful and from what I understand have helped a fair number of people integrate their non-Windows systems into Active Directory for simplified user management and authentication.  Occasionally, though, we run into the odd issue that isn’t quite so straightforward to resolve.

For example, I recently had a reader (let’s call him Johnny) who was having a difficult time getting the Linux-AD integration to work.  The “ldapsearch” and “kinit” commands worked fine, but “getent passwd” or “getent group” failed with no output.  The users in Active Directory did indeed have UNIX attributes added to their accounts.  There were no firewalls between the non-Windows systems and the Active Directory domain controllers, and there did not appear to be any connectivity issues whatsoever (this further underscored by the fact that “ldapsearch” successfully returned LDAP search results from AD, and “kinit” successfully obtained a Kerberos ticket from AD).  We were stumped.

Johnny and I traded e-mails back and forth a few times, until finally Johnny found his error and notified me about what had been happening.  As I read the description about the problem, I realized that this may be a problem that is affecting a lot of users, and may, in fact, have stumped some of you out there reading right now.  Here’s the details.

The method that I suggest using for AD integration uses two parts:

• First, we use Kerberos to obtain a Kerberos ticket from an Active Directory domain controller (also a Kerberos key distribution center, or KDC).  This handles the authentication side of things and prevents the password from crossing the wire at any point in time.
• Next, we use LDAP to centrally store account information, such as UID number, GID number, home directory, login shell, etc.  This is the part that typically requires schema extensions (although there is a workaround for that) and using this technique ensures that we don’t have to manage accounts individually on each Linux server.

This approach doesn’t work without both pieces.  The Kerberos authentication takes care of the password, but without account information logins still fail.  So if Kerberos works but LDAP doesn’t, logins will fail.  If Kerberos doesn’t work but LDAP is fine, logins will fail.  So part of troubleshooting this configuration is isolating where the problem lies.  In this particular case, “kinit” worked fine—no error was returned and “klist” showed a valid Kerberos ticket.  So the problem had to be with LDAP.  But where?  The “ldapsearch” command worked fine.

The problem lie with the /etc/ldap.conf file.  See, the nss_ldap libraries (which are responsible for using LDAP—and other sources, as defined in /etc/nsswitch.conf—as the backend information database for account information) are controlled by this file, but “ldapsearch” does not use it.  Specifically, the error was with the account that is used to bind (or connect) to Active Directory to perform the searches.

There are two ways of specifying this account in /etc/ldap.conf.  You can use the full DN, which looks something like “cn=Scott Lowe,cn=Users,dc=example,dc=com” or “cn=John Smith,ou=Marketing,ou=Departments,dc=example,dc=com”.  Alternately, you can use the universal principal name (UPN), which looks something like an e-mail address, such as “slowe@example.com” or “john.smith@example.com”.  In this particular case, Johnny (our reader with the problem) was using the full DN, but he was using the wrong attribute in the DN.  Here’s the information he had:

First Name: John
Last Name: Smith
Full Name: John Smith
Display Name: John Smith
UPN: jsmith@example.com
SAM Account Name (downlevel logon name): jsmith
Object name: jsmith

Which of these do you suppose should be used in the DN?  Full name?  No.  Display name?  No.  It must be the object name, in this case “jsmith”.  You can double-check your object name (or CN) using ADSI Edit or a similar utility.  You could use Active Directory Users and Computers, but that’s typically the confusing part.  In any case, once Johnny fixed the syntax for the bind account then “getent passwd” and “getent group” worked like a champ.

How do we avoid this kind of issue?  Simple: just use the UPN instead of the full DN.  This syntax works just as well and avoids the potential problem of using the wrong name when building the full DN.

Although much of the administration of servers running VMware ESX Server 3.0 will occur in the Windows-based Virtual Infrastructure client connected to a VirtualCenter server, there are times when it is quicker or easier to perform an administrative task directly on the ESX Server itself—either via the command-line interface (CLI) or via the VI client authenticating directly against the ESX Server.  The problem with this is that, by default, administrators will have to use different credentials when connecting the VI client to ESX Server directly.  In addition, these credentials must be managed separately from Active Directory, and separately on each individual ESX Server.  As the number of ESX Servers in a farm grows, this can quickly become an administrative nightmare.

Fortunately, we can fairly easily integrate ESX authentication into Active Directory, so that the same account used in VirtualCenter is also used when logging in to the ESX Server directly.  While VMware took the step of automating a portion of this process for us in the esxcfg-auth command, it only takes us part of the way.

Let’s take a look at what part esxcfg-auth does accomplish for us, then look at how to accomplish the rest of the task.

Using esxcfg-auth

The esxcfg-auth command will help automate a good portion of the work required to integrate ESX Server into Active Directory; specifically, it will automate the Kerberos configuration.  To use esxcfg-auth to enable AD authentication, use the following command (lines are wrapped for readability; the backslash indicates line continuation):

esxcfg-auth --enablead --addomain=example.com 
–addc=dc1.example.com

Obviously, you’ll want to substitute the appropriate values for “example.com” and “dc1.example.com” on the command above.  So what does this command do, exactly?  Here’s the breakdown:

• Modifies the /etc/krb5.conf file to use example.com as the default Kerberos realm, and to use dc1.example.com as a KDC for that realm.  In this same file, the domain “example.com” is mapped to the realm “EXAMPLE.COM” (keep in mind Kerberos realms are always specified in UPPERCASE).
• The /etc/pam.d/system-auth file is modified to use pam_krb5.so for Kerberos authentication.

What does this command not do?  Well, for one it doesn’t configure the ESX Server for anything other than pure authentication.  This means that although users will be forced to authenticate against Active Directory, ESX Server still expects to find the accounts defined in the local /etc/passwd file.  This means that password management is centralized, but account management is still decentralized.  (Some might see this as a security advantage, in that we must manually define an account in order to allow that account to log in to that server.)

To fully centralize account management, we’ll need to step outside of the esxcfg-auth framework and get our hands dirty.  Ready?

Finishing esxcfg-auth’s Work

To fully round out the authentication/account management configuration, Active Directory will have to be made aware of some UNIX-specific attributes.  This means extending the schema.  If you are running Windows 2000 or Windows Server 2003 pre-R2, this means installing Services for UNIX (SfU) 3.5; for Windows Server 2003 R2 or later, this means installing Identity Management for UNIX.

I’ll refer you to this article for Windows 2003 and Windows Server 2003 pre-R2 and this article for Windows Server 2003 R2 or later for more information.  (Because the ESX Server service console is based on Red Hat Enterprise Linux, these Linux-AD integration guides are very applicable here.)

Once Active Directory has been configured, the only tasks left for us to do are 1) to configure the nss_ldap libraries; 2) to configure /etc/nsswitch.conf to enable LDAP for naming services; and 3) verify time synchronization.

To configure the nss_ldap libraries, you’ll need to modify the /etc/ldap.conf file as described in Step 5 of the “Prepare Each Linux Server” section of this article (assuming you are using Windows Server 2003 R2).  This sets up the connection to Active Directory via LDAP and configures the attribute maps accordingly.

Please note that you’ll need to create an account in Active Directory (a Domain Guest is fine) that the nss_ldap libraries may use for LDAP queries.  You’ll specify that account information when configuring /etc/ldap.conf.

Next, you’ll need to configure /etc/nsswitch.conf.  You only need to modify the passwd, group, and shadow lines, and you only need to add “ldap” at the end of the lines.  The lines will end up looking something like this:

group:    	files ldap
passwd:    	files ldap
shadow:    	files ldap

At this point, you should be able to test the nss_ldap configuration.  Run “getent passwd <AD user account>” and you should get back information about that account’s home directory, login shell, UID, and name.  If you don’t get back any information, go back and double-check your configuration.

To verify time synchronization, have a look at the NTP configuration found in /etc/ntp.conf.  Make the changes you need here to be sure that both Active Directory (the forest root PDC emulator, specifically) and ESX Server are both synchronizing time and will stay in sync.  Otherwise, the Kerberos authentication process will fail.  Keep in mind that you may need to adjust the Service Console firewall using esxcfg-firewall in order to allow the appropriate traffic outbound.  (Thanks to KentA for reminding me about this step!)

Once “getent passwd” is working as expected and time is in sync, then you should be able to SSH into the ESX Server with any appropriately configured AD account (i.e., any AD account that has the “UNIX Attributes” tab completed with valid information).  This gives you a similar level of control over who is allowed to login and who isn’t; accounts that don’t have any UNIX attributes won’t be able to authenticate and gain access to the ESX Servers.

In addition, you should be able to configure some level of access control as described here.

Summary

To summarize, the process for integrating ESX Server into Active Directory looks like this:

1. Use esxcfg-auth to set up the Kerberos authentication.
2. Extend the AD schema (if necessary) to include UNIX attributes such as login shell, UNIX home directory, UID, UID number, etc.  This is accomplished in different ways depending upon the version of Windows in use.
3. Populate the UNIX attributes for those user accounts that should be allowed to access the ESX Server(s).
4. Configure /etc/ldap.conf on the ESX Server to configure LDAP connectivity back to Active Directory for name service lookups.
5. Configure /etc/nsswitch.conf to use LDAP for name service lookups.
6. Verify (and configure, if needed) time synchronization via NTP on the ESX Server.
7. Test the configuration using “getent passwd” or “getent group” or both.

This configuration will centralize not only authentication for the ESX Servers but will also centralize account management in Active Directory as well.

Please feel free to add any corrections or suggestions for improvement in the comments below.  Thanks!