Interoperability

I’ve received some feedback from a reader who alerted me to some sort of interaction between the Local Security Policy on the Windows side and Linux servers authenticating to Active Directory via Kerberos/LDAP/Samba.  I haven’t quite been able to get to the root issue yet, but here’s the high level overview.

The reader was seeing strange delays at the end of a Linux logon process that seemingly could not be explained.  After jumping through all the hoops, another administrator within the organization changed the Local Security Policy setting that governed the use of LM and NTLM authentication, and the delays disappeared.

The policy had been set to allow both LM and NTLM authentication; when changed to allow only NTLM authentication, the delays disappeared immediately.  The Linux server in question did have Samba installed, so apparently Samba was timing out trying the LM authentication; this caused the delays.  Of course, this is all just speculation, as we don’t know exactly why the policy change eliminated the delay.

In any case, since I’ve been pushing the use of Samba in my latest integration instructions (Solaris version here), I thought it might be prudent to mention this feedback.  In the event you start seeing some strange delays in your Linux authentication requests, check the Local Security Policy and see if LM authentication is being permitted.  That might just be your culprit.

Regular readers of this blog know that I like to work on integrating various systems into Active Directory.  I’ve written a couple of articles on the issue:

Linux-AD Integration, Version 4
Solaris 10-AD Integration, Version 3
Active Directory Integration Index

These articles have been pretty successful and from what I understand have helped a fair number of people integrate their non-Windows systems into Active Directory for simplified user management and authentication.  Occasionally, though, we run into the odd issue that isn’t quite so straightforward to resolve.

For example, I recently had a reader (let’s call him Johnny) who was having a difficult time getting the Linux-AD integration to work.  The “ldapsearch” and “kinit” commands worked fine, but “getent passwd” or “getent group” failed with no output.  The users in Active Directory did indeed have UNIX attributes added to their accounts.  There were no firewalls between the non-Windows systems and the Active Directory domain controllers, and there did not appear to be any connectivity issues whatsoever (this further underscored by the fact that “ldapsearch” successfully returned LDAP search results from AD, and “kinit” successfully obtained a Kerberos ticket from AD).  We were stumped.

Johnny and I traded e-mails back and forth a few times, until finally Johnny found his error and notified me about what had been happening.  As I read the description about the problem, I realized that this may be a problem that is affecting a lot of users, and may, in fact, have stumped some of you out there reading right now.  Here’s the details.

The method that I suggest using for AD integration uses two parts:

• First, we use Kerberos to obtain a Kerberos ticket from an Active Directory domain controller (also a Kerberos key distribution center, or KDC).  This handles the authentication side of things and prevents the password from crossing the wire at any point in time.
• Next, we use LDAP to centrally store account information, such as UID number, GID number, home directory, login shell, etc.  This is the part that typically requires schema extensions (although there is a workaround for that) and using this technique ensures that we don’t have to manage accounts individually on each Linux server.

This approach doesn’t work without both pieces.  The Kerberos authentication takes care of the password, but without account information logins still fail.  So if Kerberos works but LDAP doesn’t, logins will fail.  If Kerberos doesn’t work but LDAP is fine, logins will fail.  So part of troubleshooting this configuration is isolating where the problem lies.  In this particular case, “kinit” worked fine—no error was returned and “klist” showed a valid Kerberos ticket.  So the problem had to be with LDAP.  But where?  The “ldapsearch” command worked fine.

The problem lie with the /etc/ldap.conf file.  See, the nss_ldap libraries (which are responsible for using LDAP—and other sources, as defined in /etc/nsswitch.conf—as the backend information database for account information) are controlled by this file, but “ldapsearch” does not use it.  Specifically, the error was with the account that is used to bind (or connect) to Active Directory to perform the searches.

There are two ways of specifying this account in /etc/ldap.conf.  You can use the full DN, which looks something like “cn=Scott Lowe,cn=Users,dc=example,dc=com” or “cn=John Smith,ou=Marketing,ou=Departments,dc=example,dc=com”.  Alternately, you can use the universal principal name (UPN), which looks something like an e-mail address, such as “slowe@example.com” or “john.smith@example.com”.  In this particular case, Johnny (our reader with the problem) was using the full DN, but he was using the wrong attribute in the DN.  Here’s the information he had:

First Name: John
Last Name: Smith
Full Name: John Smith
Display Name: John Smith
UPN: jsmith@example.com
SAM Account Name (downlevel logon name): jsmith
Object name: jsmith

Which of these do you suppose should be used in the DN?  Full name?  No.  Display name?  No.  It must be the object name, in this case “jsmith”.  You can double-check your object name (or CN) using ADSI Edit or a similar utility.  You could use Active Directory Users and Computers, but that’s typically the confusing part.  In any case, once Johnny fixed the syntax for the bind account then “getent passwd” and “getent group” worked like a champ.

How do we avoid this kind of issue?  Simple: just use the UPN instead of the full DN.  This syntax works just as well and avoids the potential problem of using the wrong name when building the full DN.

Although much of the administration of servers running VMware ESX Server 3.0 will occur in the Windows-based Virtual Infrastructure client connected to a VirtualCenter server, there are times when it is quicker or easier to perform an administrative task directly on the ESX Server itself—either via the command-line interface (CLI) or via the VI client authenticating directly against the ESX Server.  The problem with this is that, by default, administrators will have to use different credentials when connecting the VI client to ESX Server directly.  In addition, these credentials must be managed separately from Active Directory, and separately on each individual ESX Server.  As the number of ESX Servers in a farm grows, this can quickly become an administrative nightmare.

Fortunately, we can fairly easily integrate ESX authentication into Active Directory, so that the same account used in VirtualCenter is also used when logging in to the ESX Server directly.  While VMware took the step of automating a portion of this process for us in the esxcfg-auth command, it only takes us part of the way.

Let’s take a look at what part esxcfg-auth does accomplish for us, then look at how to accomplish the rest of the task.

Using esxcfg-auth

The esxcfg-auth command will help automate a good portion of the work required to integrate ESX Server into Active Directory; specifically, it will automate the Kerberos configuration.  To use esxcfg-auth to enable AD authentication, use the following command (lines are wrapped for readability; the backslash indicates line continuation):

esxcfg-auth --enablead --addomain=example.com \
--addc=dc1.example.com

Obviously, you’ll want to substitute the appropriate values for “example.com” and “dc1.example.com” on the command above.  So what does this command do, exactly?  Here’s the breakdown:

• Modifies the /etc/krb5.conf file to use example.com as the default Kerberos realm, and to use dc1.example.com as a KDC for that realm.  In this same file, the domain “example.com” is mapped to the realm “EXAMPLE.COM” (keep in mind Kerberos realms are always specified in UPPERCASE).
• The /etc/pam.d/system-auth file is modified to use pam_krb5.so for Kerberos authentication.

What does this command not do?  Well, for one it doesn’t configure the ESX Server for anything other than pure authentication.  This means that although users will be forced to authenticate against Active Directory, ESX Server still expects to find the accounts defined in the local /etc/passwd file.  This means that password management is centralized, but account management is still decentralized.  (Some might see this as a security advantage, in that we must manually define an account in order to allow that account to log in to that server.)

To fully centralize account management, we’ll need to step outside of the esxcfg-auth framework and get our hands dirty.  Ready?

Finishing esxcfg-auth’s Work

To fully round out the authentication/account management configuration, Active Directory will have to be made aware of some UNIX-specific attributes.  This means extending the schema.  If you are running Windows 2000 or Windows Server 2003 pre-R2, this means installing Services for UNIX (SfU) 3.5; for Windows Server 2003 R2 or later, this means installing Identity Management for UNIX.

I’ll refer you to this article for Windows 2003 and Windows Server 2003 pre-R2 and this article for Windows Server 2003 R2 or later for more information.  (Because the ESX Server service console is based on Red Hat Enterprise Linux, these Linux-AD integration guides are very applicable here.)

Once Active Directory has been configured, the only tasks left for us to do are 1) to configure the nss_ldap libraries; 2) to configure /etc/nsswitch.conf to enable LDAP for naming services; and 3) verify time synchronization.

To configure the nss_ldap libraries, you’ll need to modify the /etc/ldap.conf file as described in Step 5 of the “Prepare Each Linux Server” section of this article (assuming you are using Windows Server 2003 R2).  This sets up the connection to Active Directory via LDAP and configures the attribute maps accordingly.

Please note that you’ll need to create an account in Active Directory (a Domain Guest is fine) that the nss_ldap libraries may use for LDAP queries.  You’ll specify that account information when configuring /etc/ldap.conf.

Next, you’ll need to configure /etc/nsswitch.conf.  You only need to modify the passwd, group, and shadow lines, and you only need to add “ldap” at the end of the lines.  The lines will end up looking something like this:

group:    	files ldap
passwd:    	files ldap
shadow:    	files ldap

At this point, you should be able to test the nss_ldap configuration.  Run “getent passwd <AD user account>” and you should get back information about that account’s home directory, login shell, UID, and name.  If you don’t get back any information, go back and double-check your configuration.

To verify time synchronization, have a look at the NTP configuration found in /etc/ntp.conf.  Make the changes you need here to be sure that both Active Directory (the forest root PDC emulator, specifically) and ESX Server are both synchronizing time and will stay in sync.  Otherwise, the Kerberos authentication process will fail.  Keep in mind that you may need to adjust the Service Console firewall using esxcfg-firewall in order to allow the appropriate traffic outbound.  (Thanks to KentA for reminding me about this step!)

Once “getent passwd” is working as expected and time is in sync, then you should be able to SSH into the ESX Server with any appropriately configured AD account (i.e., any AD account that has the “UNIX Attributes” tab completed with valid information).  This gives you a similar level of control over who is allowed to login and who isn’t; accounts that don’t have any UNIX attributes won’t be able to authenticate and gain access to the ESX Servers.

In addition, you should be able to configure some level of access control as described here.

Summary

To summarize, the process for integrating ESX Server into Active Directory looks like this:

1. Use esxcfg-auth to set up the Kerberos authentication.
2. Extend the AD schema (if necessary) to include UNIX attributes such as login shell, UNIX home directory, UID, UID number, etc.  This is accomplished in different ways depending upon the version of Windows in use.
3. Populate the UNIX attributes for those user accounts that should be allowed to access the ESX Server(s).
4. Configure /etc/ldap.conf on the ESX Server to configure LDAP connectivity back to Active Directory for name service lookups.
5. Configure /etc/nsswitch.conf to use LDAP for name service lookups.
6. Verify (and configure, if needed) time synchronization via NTP on the ESX Server.
7. Test the configuration using “getent passwd” or “getent group” or both.

This configuration will centralize not only authentication for the ESX Servers but will also centralize account management in Active Directory as well.

Please feel free to add any corrections or suggestions for improvement in the comments below.  Thanks!

Samba, as I’m sure you already know, is an open source implementation of SMB/CIFS for UNIX, Linux, and similar operating systems.  I’ve found Samba to be extremely helpful in providing some assistance for integration into Active Directory, as evidenced by these articles:

• Solaris 10-AD Integration, Version 3
• Linux-AD Integration, Version 4

Both of these articles utilize Samba’s Active Directory support to help automate the process of joining non-Windows systems to Active Directory for the purpose of authenticating logon requests against Active Directory.

So when it came time to start working on integrating Linux or Solaris into Active Directory on Windows Server 2008, I naturally assumed that I’d be able to use Samba in the same way as I had before.  Unfortunately, that’s not the case.  Due to changes in Windows, and due to the fact that previous versions of Windows were non-standard (i.e., didn’t follow the RFCs—surprise, surprise), using Samba to join an Active Directory domain running on Windows Server 2008 currently does not work.

This thread on the Samba mailing list provides some additional information, and this Google search turns up a few hits that also provide additional information on the problem.  Until this particular issue is resolved, we won’t be able to use the “net ads join” commands (and potentially some others as well) against Active Directory domains running on Windows Server 2008.  Looks like it’s back to ktpass.exe again!

In the event that your organization is considering a migration later this year (or next?) to Windows Server 2008 (formerly “Longhorn”), here are some instructions for integrating Linux login requests against Active Directory on Windows Server 2008.  These instructions are based on Linux-AD Integration, Version 4 and utilize Kerberos, LDAP, and Samba.

When this process is complete, AD users can be enabled for use on Linux systems on the network and login to those Linux systems using the same username and password as throughout the rest of Active Directory.

If you are looking for information on using Linux with a previous version of Windows before Windows Server 2008, please refer back to my AD integration index and select the appropriate article.  The only significant changes in the process involve the mapping of the LDAP attributes; otherwise, the procedure is very similar between the two versions of Windows.

Preparing Active Directory (One-Time)

The process of installing and configuring Windows Server 2008 is beyond the scope of this article (although I may touch on that in the near future in a separate article).  Therefore, I won’t provide detailed instructions on how to perform some of these tasks, but instead provide a high-level overview.

Enable Editing/Display of UNIX Attributes

In order to store UNIX attributes in Active Directory, the schema must be extended.  To extend the schema, first install Active Directory (add the Active Directory Domain Services role to an installed server, then use the Active Directory Installation Wizard to setup Active Directory) and then add the “Identity Management for UNIX” role service (this can be done in Server Manager).

Once that role service has been installed, then the AD schema now includes a partially RFC 2307-compliant set of UNIX attributes, such as UID, UID number, GID number, login shell, etc.  (Note that it may be that these attributes are already included in the schema for Windows Server 2008; I did not check the schema before installing the Identity Management for UNIX role service.  With Windows Server 2003 R2, the schema was present at the time of installation, but the attributes were not visible until installing the UNIX identity services.)

At this point a new tab, labeled “UNIX Attributes,” will appear in the properties dialog box for users and groups in Active Directory.  You’ll use this tab to edit the UNIX-specific attributes that are required for logins to Linux-based systems.

Create an LDAP Bind Account

You’ll also need to create an account in Active Directory that will be used to bind to Active Directory for LDAP queries.  This account does not need any special privileges; in fact, making the account a member of Domain Guests and not a member of Domain Users is perfectly fine.  This helps minimize any potential security risks as a result of this account.  Just be sure that you know the account’s user principal name (UPN) and password.

Prepare Active Directory (Each User)

Each Active Directory account that will authenticate via Linux must be configured with a UID and other UNIX attributes.  This is accomplished via the new “UNIX Attributes” tab on the properties dialog box of a user account.

After all the user accounts have been configured, then we are ready to configure Active Directory objects for each of the Linux server(s) that we’ll be integrating with AD.

Prepare Active Directory (Each Server)

Prior to using Samba to join Linux computers to Active Directory and generate a keytab automatically, we had to use the ktpass.exe utility on Windows to generate a keytab.  Due to some current Samba-Windows Server 2008 interoperability issues, we can’t use Samba.  That means we’ll be back to using ktpass.exe to map service principals onto accounts in Active Directory.  Unfortunately, you’ll need to first disable User Account Control (UAC) on your server, since UAC interferes with ktpass.exe.  (Nice, huh?)

Once you’ve disabled UAC (and rebooted your server), then you can map the service principal names (SPNs) using the following steps:

1. Create a computer account (or a user account; either will work) with the name of the Linux server.
2. Use the following command to map the needed SPN onto this account (backslashes indicate line continuation):
   ktpass.exe -princ HOST/server.fqdn@REALM.COM \
-mapuser DOMAIN\AccountName$ -crypto all \
-pass Password123 -ptype KRB5_NT_PRINCIPAL \
-out filename.keytab
3. Copy this file to the Linux server (using SCP or SFTP is a good option) and merge it with the existing keytab (if it exists) using ktutil.  If there is no existing keytab, simply copy the file to /etc/krb5.keytab and you should be good to go.

Now that Active Directory has computer objects (and, more importantly, SPNs) for the Linux servers and the AD users have been enabled for UNIX (by populating the UNIX attributes), we’re ready to start configuring the Linux server(s) directly.

Prepare Each Linux Server

Follow the steps below to configure the Linux server for authentication against Active Directory.  (Note that this configuration was tested on a system running CentOS—a variation of Red Hat Enterprise Linux—version 4.3.)

1. Edit the /etc/hosts file and ensure that the server’s fully-qualified domain name is listed first after its IP address.
2. Make sure that the appropriate Kerberos libraries, OpenLDAP, pam_krb5, and nss_ldap are installed.  If they are not installed, install them.
3. Be sure that time is being properly synchronized between Active Directory and the Linux server in question.  Kerberos requires time synchronization.  Configure the NTP daemon if necessary.
4. Edit the /etc/krb5.conf file to look something like this, substituting your actual host names and domain names where appropriate:
   [logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
 
[libdefaults]
default_realm = EXAMPLE.COM
dns_lookup_realm = true
dns_lookup_kdc = true
 
[realms]
EXAMPLE.COM = {
kdc = host.example.com:88
admin_server = host.example.com:749
default_domain = example.com
}
 
[domain_realm]
.example.com = EXAMPLE.COM
example.com = EXAMPLE.COM
 
[kdc]
profile = /var/kerberos/krb5kdc/kdc.conf
 
[appdefaults]
pam = {
debug = false
ticket_lifetime = 36000
renew_lifetime = 36000
forwardable = true
krb4_convert = false
}
5. Edit the /etc/ldap.conf file to look something like this, substituting the appropriate host names, domain names, account names, and distinguished names (DNs) where appropriate.  (Please note that the nss_base_group line should not be broken across two lines when you edit it; it has been wrapped here for readability.)
   host 10.10.10.10
base dc=example,dc=com
uri ldap://server.example.com/
binddn ldap@example.com
bindpw adldapbindpw
scope sub
ssl no
nss_base_passwd dc=example,dc=com?sub
nss_base_shadow dc=example,dc=com?sub
nss_base_group dc=mydomain,dc=com?sub?
&(objectCategory=group)(gidnumber=*)
nss_map_objectclass posixAccount user
nss_map_objectclass shadowAccount user
nss_map_objectclass posixGroup group
nss_map_attribute gecos cn
nss_map_attribute homeDirectory unixHomeDirectory
nss_map_attribute uniqueMember member
6. Configure PAM (this varies according to Linux distributions) to use pam_krb5 for authentication.  Many modern distributions use a stacking mechanism whereby one file can be modified and those changes will applied to all the various PAM-aware services.  For example, in Red Hat-based distributions, the system-auth file is referenced by most other PAM-aware services.  Here’s a properly edited /etc/pam.d/system-auth file taken from CentOS 4.4 (some lines have been wrapped for readability; do not wrap them when editing the file):
   #%PAM-1.0
# This file is auto-generated.
# User changes will be destroyed the next time
# authconfig is run.
auth required /lib/security/pam_env.so
auth sufficient /lib/security/pam_unix.so
   likeauth nullok
auth sufficient /lib/security/pam_krb5.so
auth required /lib/security/pam_deny.so
 
account sufficient /lib/security/pam_unix.so
account sufficient /lib/security/pam_krb5.so
account sufficient /lib/security/pam_succeed_if.so
   uid < 100 quiet
account required /lib/security/pam_deny.so
 
password requisite /lib/security/pam_cracklib.so
   retry=3
password sufficient /lib/security/pam_unix.so
   nullok use_authtok md5 shadow
password required /lib/security/pam_deny.so
 
session required /lib/security/pam_limits.so
session required /lib/security/pam_unix.so
7. Edit the /etc/nsswitch.conf file to include “ldap” as a lookup source for passwd, shadow, and groups.

At this point we are now ready to test our configuration and, if successful, to perform the final step:  to join the Linux server to Active Directory for authentication.

Test the Configuration

To test the Kerberos authentication, use the “kinit” command, as in “kinit <AD username>@<AD domain DNS name>”; this should return no errors.  A “klist” at that point should then show that you have retrieved a TGT (ticket granting ticket) from the AD domain controller.  If this fails, go back and troubleshoot the Kerberos configuration.  In particular, if you are seeing references to failed TGT validation, check to make sure that both your Linux servers and AD domain controllers have reverse lookup (PTR) records in DNS and that the Linux server’s /etc/hosts file listed the FQDN of the server first instead of just the nodename.

<aside>Some readers and some other articles have suggested the use of the AD domain DNS name in the /etc/krb5.conf file instead of an AD domain controller specifically; I recommend against this.  First, I believe it may contribute to TGT validation errors; second, it is possible to list multiple KDCs (AD DCs) in the configuration.  Since the only major reason to use the AD domain DNS name instead of the DNS name of one or more DCs would be fault tolerance, then it doesn’t really gain anything.</aside>

To test the LDAP lookups, use the “getent” command, as in “getent passwd <AD username>”; this should return a listing of the account information from Active Directory.  If this does not work, users will not be able to login, even if Kerberos is working fine.  If you run into errors or failures here, go back and double-check the LDAP configuration.  One common source of errors is the name of the LDAP bind account, so be sure that is correct.

At this point, SSH logins to the Linux system using an account present in Active Directory (one which has had its UNIX attributes specified properly) should be successful.  This will be true as long as you used the ktpass.exe command earlier to map the SPN onto the computer object in Active Directory.  Even if you didn’t copy the keytab over to the Linux server, logins will work.  Why?  Because the PAM Kerberos configuration, by default, does not require a client keytab, and does not attempt to validate the tickets granted by the TGT.  This means that as long as the SPN(s) are mapped to the accounts in AD, the keytab is not necessarily required.

(Note, however, that not using a keytab and/or not requiring a keytab does leave the Linux server open to potentially spoofed Kerberos tickets from a fake KDC.  In addition, “native” Kerberos authentication—i.e., using a Kerberos ticket to authenticate instead of typing in a password—won’t work without a keytab.)

Deal with Home Directories

Unlike Windows systems, home directories are required on Linux-based systems.  As a result, we must provide home directories for each AD user that will log in to a Linux-based system.  We basically have three options here:

• Manually create home directories and set ownership/permissions properly before users will be able to log in.
• Use the pam_mkhomedir.so PAM module to automatically create local home directories “on the fly” as users log in.  To do this, you would add an entry for pam_mkhomedir.so in the session portion of the PAM configuration file.
• Use the automounter to automatically mount home directories from a network server.  This process is fairly complicated (too involved to include the information here), so I’ll refer you to this article on using NFS and automounts for home directories.  This has the added benefit of providing a foundation for unified home directories across both Windows and Linux systems.

Once you’ve settled on and implemented a system for dealing with home directories, you are finished!  UNIX-enabled users in Active Directory can now login to Linux-based systems with their Active Directory username and password.

What’s not addressed in this article?  Password management.  In this configuration, users will most likely not be able to change their password from the Linux servers and have that change properly reflected in Active Directory.  In addition, “native” Kerberos authentication using Kerberos tickets won’t work unless the keytab is present.  In my testing, I ran into a number of issues with the keytab and TGT validation, but I’m not sure if those are errors in my process or the result of the beta status of Windows Server 2008.

I welcome your corrections, additions, or suggestions in the comments below.

The following configuration will enable you to authenticate login requests to Cisco equipment running IOS against Active Directory.  This would, for example, allow you to centralize the authentication of your Cisco-based network infrastructure against Active Directory.

Configuring the Cisco Equipment

The equipment I used in this configuration was a Cisco Catalyst 3560G switch running IOS 12.2(25); please note that the commands listed here may be different in different versions of IOS.  The commands should be roughly equivalent, however, across hardware platforms.

First, we must enable the external authentication mechanisms, then we’ll specify the external authentication servers we’re going to use.  This is listed below:

1. First, to enable external authentication on the switch, use the following commands in global configuration mode:
   s1(config)#aaa new-model
s1(config)#aaa authentication login default group radius local
   This enables the authentication of login requests by RADIUS first, then by a local database (just in case network connectivity is down).  We specify “local” as well because this configuration applies to both telnet requests as well as physical console requests.
2. Next, we specify the external authentication servers that the switch should use:
   s1(config)#radius-server host 10.1.1.254 auth-port 1645
acct-port 1646 key Password
   (This should all be on one line.)  Best practices dictate that you should have at least two RADIUS servers for redundancy.  Note that the “auth-port” and “acct-port” parameters are only necessary if you are using nonstandard ports.  Since Microsoft’s IAS (Internet Authentication Service, which provides the RADIUS interface to Active Directory) uses both sets of standard ports (1645/1812 and 1646/1813) you won’t need to specify these parameters.  The “key” parameter is a shared secret key between the RADIUS client (the switch) and the RADIUS server.  Obviously, you’ll want to use something other than “Password”.

Now we’re ready to move to configuring the Windows servers that we’ll use for RADIUS authentication.

Configuring Internet Authentication Service (IAS)

Configuring IAS is rather simple.  I’ve discussed the use of IAS before (here in discussing Cisco PIX-AD integration and here regarding WatchGuard Firebox-AD integration), and I’ll refer you back to those articles for some of the basics on setting up and configuring IAS.

Note that these instructions are based on the version of IAS included with Windows Server 2003 R2; different versions may behave slightly differently.

To configure IAS in this instance (once it has been installed and registered with Active Directory), we’ll do the following:

• Add the Cisco Catalyst switch as a RADIUS client.  We’ll need to be sure to specify the same shared secret as used in the switch configuration above.  You can specify the Cisco switch either by DNS name (if it is registered in DNS) or by IP address.
• Create a new remote access policy that grants remote access permission.  The conditions on the policy should be “NAS-IP-Address” (set to the IP address of the Cisco equipment) and “Windows-Groups” (set to whatever group should be allowed to authenticate to the switch; I created a group called “Cisco Admins” and used it).
• Configure the profile to use only PAP authentication and no encryption.

Repeat this process on the second Windows server running IAS (you did configure two for redundancy, didn’t you?).

That’s it!  At this point, you should be able to telnet to the Cisco switch (or whatever IOS-based equipment you’ve configured) and log in with your Active Directory username and password.  Once logged in, you can use your enable or enable secret password to enter privileged exec mode.

Now, before you go any farther, add a local account to use in case the network connectivity to the RADIUS server is lost:

s1(config)#username localaccount password password123

(Obviously, you’ll want to use a secure password!)  This will ensure that if you lose network connectivity to the equipment, you can still get in through the serial console connection.  Be warned: without this local account, you can be locked out of the equipment completely if the RADIUS server(s) are inaccessible!

This Cisco document offers some additional information on AAA configurations, so I’ll refer you there for more detailed descriptions of the commands involved.  Enjoy!

Network Appliance storage systems support the use of virtual interfaces (VIFs) to provide link redundancy and improved network throughput.  Two types of VIFs are available:

• Single-mode VIFs act like a fault tolerant team and will fail traffic over to a standby link when the active link goes down.
• Multi-mode VIFs act like a group of links providing aggregate bandwidth as well as link redundancy.

Single-mode VIFs are great for fault tolerance, but the storage system isn’t leveraging all the links.  It’s “active-passive” arrangement in which only one of the links is passing traffic while the other link is idle.  No switch support is required for this configuration.

Multi-mode VIFs, on the other hand, allow for both greater bandwidth utilization as well as fault tolerance.  Traffic will be distributed across all the links in the VIF (typically based on IP address), and if one link fails the traffic is redistributed across the remaining links.  However, this configuration requires support on the switch.  In this article, we’re going to look at configuring a Cisco Catalyst 3560 switch to do link aggregation with a NetApp storage system running Data ONTAP 7.1.1.1.

To configure the switch, we’ll use the following commands (these are entered in global configuration mode on the switch):

s3(config)#int port-channel1
s3(config-if)#description Multi-mode VIF for netapp1
s3(config-if)#int gi0/23
s3(config-if)#channel-group 1 mode on
s3(config-if)#int gi0/24
s3(config-if)#channel-group 1 mode on

This creates the port-channel1 interface (you may need to increment that number, i.e., use port-channel2 or port-channel3, if you already have existing link aggregates configured) and adds interfaces GigabitEthernet0/23 and GigabitEthernet0/24 to the link aggregate.  If you do have to use a different link aggregate interface, be sure the number of the interface (“int port-channel4”) matches the number of the channel-group specified on the member interfaces (“channel-group 4 mode on”).  This seems obvious, but it’s worth mentioning nevertheless.

Be aware that Data ONTAP’s multi-mode VIFs are only compatible with static 802.3ad link aggregation; you can’t use PAgP (Cisco proprietary protocol).  I would assume dynamic LACP is also incompatible.  For this reason we used the “channel-group 1 mode on” statement instead of something like “channel-group 1 mode desirable”.

By default, many Cisco switches default to MAC address-based load balancing across the links, whereas NetApp defaults to IP address-based load balancing.  To see the switch’s current load balancing configuration, use this command in privileged mode:

s3#show etherchannel load-balance

To change the switch’s load balancing algorith to a mode compatible with NetApp’s, use one of the following command in global configuration mode (note that changing it affects the entire switch; you can’t change it for a single port-channel individually):

s3(config)#port-channel load-balance src-dst-ip

Once the switch is configured, then we can proceed with configuring the NetApp storage system.  The following commands will create the the multi-mode VIF (this can also be done via the FilerView GUI):

netapp1>vif create multi vif0 e6d e7d
netapp1>ifconfig vif0 172.31.254.10 netmask 255.255.255.0
netapp1>ifconfig vif0 up

This creates the VIF with interfaces e6d and e7d as members, plumbs it with an IP address, and brings it up.  Running the command “vif status vif0” now will return the following results:

default: transmit ‘IP Load balancing’, VIF Type ‘multi_mode’, fail ‘log’
vif0: 2 links, transmit ‘IP Load balancing’, VIF Type ‘multi-mode’ fail ‘default’
 
VIF Status Up Addr_set
up:
e6d: state up, since 05Oct2001 17:17:15 (05:23:05)
mediatype: auto-1000t-fd-up
flags: enabled
input packets 2000, input bytes 12800
output packets 173, output bytes 1345
up indications 1, broken indications 0
drops (if) 0, drops (link) 0
indication: up at boot
consecutive 3, transitions 1
e7d: state up, since 05Oct2001 17:18:03 (00:10:03)
mediatype: auto-1000t-fd-up
flags: enabled
input packets 134, input bytes 987
output packets 20, output bytes 156
up indications 1, broken indications 0
drops (if) 0, drops (link) 0
indication: broken

Note the ‘IP Load balancing’ algorithm stated in the output; this is why the switch’s load-balancing mechanism should be changed to match.

At this point, the links should be up between the Cisco switch and the NetApp storage system, and traffic should be passing to and from the storage system without any problems.  To test the fault tolerance, we can pull one of the links in VIF; traffic should continue to flow with very little, if any, interruption.  And while traffic from a single client to the NetApp won’t see a significant increase in throughput, the overall throughput of multiple separate clients to the NetApp should improve with multiple links in the VIF.

More information, including additional Cisco configs, is available here.

There’s a small gotcha when using Microsoft’s iSCSI initiator and MPIO driver to do iSCSI multipathing:  the Microsoft initiator and MPIO driver will overwrite the IQN of the iSCSI HBA.  Obviously, this could cause problems where access control to iSCSI LUNs is based on initiator IQN.

As pointed out in this Qlogic support document (check the “Additional Notes” section at the bottom of the page), installation of the Microsoft iSCSI initiator will overwrite the IQN of the HBA with a Microsoft-generated IQN, like “1991-05.com.microsoft:servername.domain.com” or similar.

In environments where access to LUNs is controlled in part or in whole by initiator IQN, this is a problem.  One such environment is NetApp iSCSI SANs, where initiator groups (or “igroups”) control access to LUNs based on the IQNs of the initiators.  To work around this, you’ll want to add the original IQNs of the HBAs (before the installation of the Microsoft iSCSI initiator) as well as the Microsoft IQN in the igroups for the LUNs that should be visible to that server.  Otherwise, you could lose access to the LUN after installation of the Microsoft initiator.

(By the way, in case you’re wondering why one would install the Microsoft iSCSI initiator when you’ve already got HBAs, there’s a good reason—to get multipath support.)

Thanks to some very helpful individuals in the #solaris channel on irc.freenode.net, I’ve been able to get ADS support working in Samba on Solaris 10, and thus have been able to incorporate the use of Samba in the Solaris 10-AD integration instructions.

To refer to earlier versions of the Solaris 10-AD integration instructions, see this article or this article.  I would expect that you won’t need to refer to those posts, though, and will be able to get most of what you need directly from this post.

Assumptions

This procedure assumes that you are using Windows Server 2003 R2; if you are using a previous version, the LDAP attribute mapping will need to be modified to match the schema extensions found in Microsoft’s Services for Unix (SfU) add-on product.  This will require changes to the “ldapclient manual” command shown below, which handles the schema/attribute mapping.  (I only have a single article written that includes pre-R2 attribute mapping, and that’s this Linux-AD article.  The schema mapping should be very, very similar between that article and Solaris 10.)

Preparing Active Directory (One-Time)

These steps only need to be performed once.  Note that if you have performed any of these steps as part of authenticating Linux or Solaris to Active Directory, they do not need to be performed again.  Simply make note of the information used earlier and re-use that information again this time.

1. Install the “Server for NIS” component on at least one Active Directory domain controller (DC), so that the Active Directory schema can be extended to become partially RFC 2307-compliant.  Installing this component will also add a “UNIX Attributes” tab to objects inside the Active Directory Users and Computers MMC console.  You may also need to install the Server for NIS administrative tools on your workstation to see the “UNIX Attributes” tab.
2. Use the Schema Management MMC snap-in to index the uid attribute, which is not indexed by default.  This will speed up the login process and reduce the overall load on your DCs.  (For more information, refer to the Linux-Windows Server 2003 R2 integration instructions.)  It may be possible to change the attribute that Solaris is looking for, but I haven’t found a way to do that yet.
3. Create an account in Active Directory that will be used to bind to Active Directory for LDAP queries.  This account does not need any special privileges; in fact, making the account a member of Domain Guests and not a member of Domain Users is perfectly fine.  I recommend giving this account a simple, short name; this will make specifying the DN of the account later easier to do.
4. Create a global security group in Active Directory Users & Computers and set the UNIX attributes for this group.

Once these one-time steps have been completed, we can proceed to configuring the individual users that will be authenticating to Active Directory from your Solaris server(s).

Preparing Active Directory (Each User)

Each Active Directory account that will authenticate via Solaris must be configured with a uid and other UNIX attributes.  This is accomplished via the new “UNIX Attributes” tab on the properties dialog box of a user account (this tab was made visible by the installation of the Server for NIS component).  The attributes that must be populated are:

• NIS domain:  It’s required on this tab in order to populate the other fields, but we won’t be using it.
• UID:  This is actually the UID number.  Each user must have a unique UID; I believe that the Server for NIS defaults at a starting UID of 10000, which is pretty safe for most systems.
• GID:  In addition, each member must have a GID (group ID); simply specify the group that was created earlier.
• Login Shell:  Specify a login shell (such as “usr/bin/csh” or “/sbin/sh”) for each user.
• Home Directory:  Specify the home directory (such as “/export/home/slowe”) that will be used for this user.  Keep in mind that these values may apply across multiple systems and platforms, and the path must be valid on all systems and platforms.

Based on my experience so far, the values for Solaris will often be very different than what might be specified for Linux-based logins.  The only workarounds I’ve found to address these issues is the clever use of symlinks and the use of NFS automounts for home directories.

After all the user accounts have been configured, then we are ready to perform the additional tasks within Active Directory and on the Solaris server(s) that will enable the authentication.

Configuring Reverse DNS

On the DNS server handling the reverse lookup zones for the subnet on which the Solaris server is located, add a PTR record for the Solaris server and it’s IP address.  This will ensure that reverse DNS lookups work as expected.  Make sure that each Solaris server that will be authenticating against Active Directory has a reverse lookup record in DNS, and ensure that both forward and reverse lookups work from each of the Solaris server(s).

Configuring Solaris (Each Server)

The following steps need to be performed on each Solaris server that will authenticate against Active Directory.

Configuring the hosts file

To enable reliable TGT validation (this ensures that the Kerberos ticket returned by a KDC actually came from the KDC and not a spoofed server), you’ll need to edit the hosts file.  On Solaris 10, this is found in /etc/inet/hosts and is read-only, even for root.  Edit this file (changing permissions as necessary) so that the line with the server’s IP address looks something like this:

10.1.1.1        hostname.example.com hostname loghost

What we’re doing here is making sure that the server’s fully qualified domain name (not just the short hostname) is the first name entry on the line for the server’s IP address.

There may or may not be other entries in the file; leave those entries untouched (unless you know you need to modify them).

Installing Blastwave Packages

This is the key to getting ADS support into Samba on Solaris 10.  I won’t go into excruciating detail on this since this process is amply covered elsewhere, but here’s the basic idea of the process:

• Use the standard wget (found in /usr/sfw/bin) to download the pkg-get file used by Blastwave.
• Use pkgadd to install pkg-get.
• Configure pkg-get to use the unstable packages (makes sure you get the latest builds).
• Use pkg-get to install the CSWsamba package and all requisite packages (there were quite a few dependency packages during my testing).

Once the CSWsamba package and related packages are installed, we’ll need to configure Samba by creating /opt/csw/etc/samba/smb.conf with the following contents:

workgroup = <NetBIOS name of AD domain>
security = ads
realm = <DNS name of AD domain in UPPERCASE>
use kerberos keytab = true
password server = <Space-delimited list of AD DCs>

At this point, we are ready to configure Kerberos and then proceed with testing the configuration and join the Active Directory domain.

Configuring Kerberos

Solaris keeps its Kerberos configuration in the /etc/krb5 directory as krb5.conf.  Edit this file using your editor of choice to look something like the one below.  Depending upon how you configured Solaris during the installation, some of this configuration may already be present.

[libdefaults]
        default_realm = EXAMPLE.COM
        dns_lookup_kdc = true

[realms]
        EXAMPLE.COM = {
        kdc = dc01.example.com
        kdc = dc02.example.com
        admin_server = dc01.example.com
        }

[domain_realm]
        .example.com = EXAMPLE.COM
        .subdomain.example.com = EXAMPLE.COM

[logging]
        default = FILE:/var/krb5/kdc.log
        kdc = FILE:/var/krb5/kdc.log
        kdc_rotate = {
        period = 1d
        version = 10
        }

[appdefaults]
        kinit = {
        renewable = true
        forwardable= true
        }

There will also be a file named cswkrb5.conf in the /etc directory; you can configure this file with the contents of the [libdefaults], [reamls], and [domain_realms] sections as listed above.  You don’t need to include the [logging] or [appdefaults] sections in this file.

Note that you can’t simply copy and paste from here to the Solaris configuration files, as you’ll need to customize your version for your particular network, hostnames, domain names, etc.  If you must copy and paste from here, put it into a text editor first to customize it for your implementation.

Configuring LDAP

We’ll use the native Solaris “ldapclient” utility to configure the LDAP support in Solaris.  The command you’ll type in looks something like this (please don’t copy and paste this, as it contains generic/incorrect information that won’t work!):

ldapclient manual \
-a credentialLevel=proxy \
-a authenticationMethod=simple \
-a proxyDN=cn=proxyuser,cn=Users,dc=example,dc=com \
-a proxyPassword=Password1 \
-a defaultSearchBase=dc=example,dc=com \
-a domainName=example.com \
-a “defaultServerList=172.16.1.10” \
-a attributeMap=group:userpassword=userPassword \
-a attributeMap=group:memberuid=memberUid \
-a attributeMap=group:gidnumber=gidNumber \
-a attributeMap=passwd:gecos=cn \
-a attributeMap=passwd:gidnumber=gidNumber \
-a attributeMap=passwd:uidnumber=uidNumber \
-a attributeMap=passwd:homedirectory=unixHomeDirectory \
-a attributeMap=passwd:loginshell=loginShell \
-a attributeMap=shadow:shadowflag=shadowFlag \
-a attributeMap=shadow:userpassword=userPassword \
-a objectClassMap=group:posixGroup=group \
-a objectClassMap=passwd:posixAccount=user \
-a objectClassMap=shadow:shadowAccount=user \
-a serviceSearchDescriptor=passwd:dc=example,dc=com?sub \
-a serviceSearchDescriptor=group:dc=example,dc=com?sub

The easiest way to handle this would probably be to copy it into a blank text file, edit it to include the specific details for your network, and then paste it into a terminal session on the Solaris server.

After this command has been run, Solaris will create the LDAP configuration in /var/ldap and will update /etc/nsswitch.conf to use LDAP.  However, because we only want to use LDAP for specific purposes, we’ll need to go back and edit /etc/nsswitch.conf again.  Just remove “ldap” from all entries in /etc/nsswitch.conf except for passwd and group.

While you’re editing /etc/nsswitch.conf, be sure to add a “dns” entry at the end of the line for hosts:

hosts          files dns

This will help ensure that Solaris is properly configured to use DNS for host name resolution.

I think it’s necessary at this point to restart the LDAP client service:

svcadm restart svc:/network/ldap/client:default

Use the “svcs -a | grep ldap” command to verify the exact name of the LDAP client service on your Solaris server.

Configuring the DNS Client

You’ll also need to make sure that the DNS client is enabled and running.  Using “svcs -a | grep dns” will help you identify the correct service, which you can then enable with svcadm:

svcadm enable svc:/network/dns/client:default

Test DNS resolution using the “nslookup” command.  As mentioned previously, be sure to test both forward and reverse lookups.

Configuring PAM

The /etc/pam.conf file controls the PAM (Pluggable Authentication Mechanism) configuration on Solaris.  You’ll need to edit the /etc/pam.conf file to look something like what’s shown below.  I’ve edited away all the non-essential sections, so only change the sections listed below.

# Default definition for Authentication management
#
other   auth requisite          pam_authtok_get.so.1
other   auth required           pam_dhkeys.so.1
other   auth sufficient         pam_krb5.so.1
other   auth required           pam_unix_cred.so.1
other   auth required           pam_unix_auth.so.1
#
# Default definition for Account management
#
other   account requisite       pam_roles.so.1
other   account sufficient      pam_unix_account.so.1
other   account required        pam_ldap.so.1
#

With this configuration in place, Solaris will use Kerberos authentication and will retrieve account information via LDAP.

Reboot the Solaris Server

I know this sounds stupid, but even after restarting LDAP and enabling/starting/restarting the DNS client, things still didn’t work for me in the lab.  However, after rebooting the Solaris server, it worked like a champ.  So, just in case, reboot the Solaris server after completing the configuration.

Testing the Configuration

Once all of the configuration steps have been completed, you can test the configuration with the following commands:

• You can use “getent passwd <Name of AD user>” from the Solaris server; this command should return UID number, GID number, UNIX home directory, and login shell.
• You can use “kinit <Name of AD user>” to test Kerberos authentication.  A succesful Kerberos test will not return any feedback, and the “klist” command will show a ticket granting ticket (TGT) from the Active Directory DC/KDC.

If either of these tests are unsuccessful, review the log files on the Solaris server and resolve the problems before continuing.  Both of these tests will need to be successful in order for authentication to work correctly.

If the tests are successful, then you should now be able to join the Solaris server to Active Directory using Samba.

Joining the Solaris Server to Active Directory

This is the final step.  Don’t try this step until you’ve successfully tested the configuration.  After this step is completed, you are finished and AD users will be able to login to the Solaris server (assuming the AD users have been properly configured).

To join the Solaris server to Active Directory, follow these steps:

1. Verify the Samba configuration as outlined earlier.  Key to the configuration are the “security = ads” and “use kerberos keytab = true” directives.
2. Use “kdestroy” to destroy any existing Kerberos credentials you may have; then run “kinit <Domain administrative account>@AD.DOMAIN.NAME” to get a Kerberos ticket for an account that is a domain administrator account.
3. Run “net ads join” to join the Solaris server to Active Directory.  This command will automatically create a computer object in Active Directory and add the appropriate SPNs (service principal names) to the computer object.  In addition, it will populate the local Kerberos key table (/etc/krb5.keytab, by default) with the correct entries for authentication against Active Directory.  You may see an error about a missing userPrincipalName, but this does not appear to affect any functionality.

At this point, all properly configured AD users (those users who have UNIX attributes) should be able to login to the Solaris server using their Active Directory username and password.  Of course, this assumes that you’ve already dealt with home directories (or are automounting home directories).

As with previous instructions, these instructions don’t address password management (the ability to change an AD password from Solaris) and don’t address how to handle home directories.  Hey, I’ve got to leave a few challenges for others to tackle, right?

How I Tested

This testing was done using Solaris 10 11/06 (Update 3) running on VMware ESX Server 3.0.1.  Active Directory was running on a pair of servers with Windows Server 2003 R2, also virtual machines on ESX Server.  Authentication testing was performed using SSH from a client system running Mac OS X.

The last update to the Solaris 10-Active Directory integration instructions was in October of last year, over six months ago.  Since that time, Sun has released another update to Solaris (Solaris 10 11/06, or Update 3) and I have been able to gather some additional information on using an Active Directory-aware version of Samba to help with the process (much like described in the latest version of the Linux-AD instructions).

The new version will use Kerberos for authentication, LDAP for account information, and Samba to do the “heavy lifting” of joining Active Directory, creating the necessary objects, and creating the keytab and keytab entries on Solaris.

I hope to post the updated integration instructions within the next few days, before I have to leave for a business trip to Canada.