There was a clear correlation between the DR DC and the AD-integrated Linux systems, even though the /etc/ldap.conf file specifically pointed to another DC by IP address. There was no reference whatsoever, by IP address or host name, to the DR DC. Yet, every time the DR DC was taken offline, the behavior returned on a subset of Linux hosts. The only difference we could find between the affected and unaffected hosts was that the affected hosts were not on the same VLAN as the production domain controllers.

I theorized that Windows’ netmask ordering feature, which prioritizes the return of DNS lookups to provide clients with addresses that are “closer” to them, was playing a role here. However, the /etc/ldap.conf was using IP addresses, not the domain name or even the fully qualified domain name of a DC. It couldn’t be DNS, at least not as far as I could tell.

Upon further investigation, the reader discovered that the affected Linux servers—those that were on a different VLAN than both the production DCs as well as the DR DC—were maintaining persistent connections to the DR DC. (He found this via netstat.) When the DR DC went offline, the affected Linux hosts tried to continue to communicate to that DC and that DC only. Once the reader was able to get the affected Linux hosts to drop that persistent connection, he was able to take the DR DC offline and the Linux hosts worked as expected.

So, the real question now becomes: how (or why) did the Linux servers connect to the DR DC instead of the production DC for which they were configured? I think that Active Directory issued an LDAP referral to direct the affected Linux servers to the DR DC as a result of site topology. Perhaps due to an incorrect or incomplete site topology configuration, Active Directory believed the DR DC should handle the VLANs where the affected Linux servers resided. If that is indeed the case, the fix would be to make sure that your AD site topology is correct and that subnets are appropriately associated with sites. Of course, this is just a theory.

Has anyone else seen an issue similar to this? What fix were you able to implement in order to correct it?

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

Linux-AD Integration Issue: Persistent Connections

• ESX Server-AD Integration
• Local Logins Refused in AD Integration Scenarios
• Linux-AD Integration with Windows Server 2008
• Using Samba in Linux-AD Integration
• Performance in Linux-AD Integration Scenarios

My primary goals for this “home network rebuild” were the following:

1. Rebuild the home server with a newer version of Linux, and possibly switch to a different distribution.
2. Continue to provide DNS, DHCP, HTTP, and HTTP proxying/content filtering services to the home network.
3. Continue to provide file sharing services via Server Message Block/Common Internet File System (SMB/CIFS) for Windows-based systems on the home network.
4. Continue to have a shared music library available via Digital Audio Access Protocol (DAAP, aka iTunes) available to all systems on the home network.
5. Provide file sharing services to Macs on the network via AppleTalk Filing Protocol (AFP) over TCP.

Ideally, I also wanted to enable Time Machine backups from my Mac laptop to the home server.

After doing a fair amount of research, I settled on the use of Ubuntu 8.04 LTS (”Hardy Heron”) for the server build. I didn’t go with Ubuntu 8.10 (”Intrepid Ibex”) simply because a) I already had 8.04.1 downloaded and burned to a CD; and b) Hardy Heron is an LTS release, so I should have better support over the long term.

I won’t bore readers with the details of the rebuild, but it took about a day or two to get a larger hard drive installed, Ubuntu installed and configured, and services running like DHCP (including some static reservations for certain computers, like my laptop and my iPhone), DNS (using MaraDNS, much easier to figure out than BIND), Apache, and Squid with SquidGuard. At this point, I’d completed tasks #1 and #2.

On to task #3. This was pretty simple and straightforward and easily accomplished via Samba, with nothing really unique to document here. The one interesting thing that I did find was a way to map the long usernames that Mac OS X uses (like “Bob Jones”) to a short username (like “bjones”). I used this command in the /etc/samba/smb.conf file:

username map = /etc/samba/usermap.conf

In this file, I simply placed lines that mapped the long usernames to the short usernames. Since Mac OS X defaults to the long username when connecting to the server, this allows me to simply type in a password and connect. I searched for hours trying to find a way to have Mac OS X supply my current password to the Samba server so that I wouldn’t get prompted, but could not find any information. If anyone knows the trick, I’d love to hear about it. After configuring a few shares, setting Linux permissions and the umask, and then testing from both my Mac laptop and a Windows laptop, task #3 was finished.

Task #4, providing an iTunes-compatible music server, was also really straightforward and easy. For this, I again selected Firefly Media Server, formerly mt-daapd, which I’d used before with great success. Again, nothing unusual or unique to document here, except for the potential interaction with Avahi (more on that later).

The final task was installing Netatalk to provide AFP over TCP file sharing services for Macs on the network. Fortunately for me, one of the sites I’d been using to help in my project pointed me to this blog post, which had a prebuilt package for Netatalk that included the necessary SSL support that Mac OS X requires. That saved me the trouble of compiling Netatalk from source. Following the steps in the Kremalicious article as well as information from this guide, I configured Netatalk to present a volume to use for Time Machine backups. It was at this point that I noticed a strange interaction with Avahi.

Avahi is a multicast DNS (what Apple calls Bonjour) server for Linux. It’s responsible for advertising services to multicast DNS-enabled systems, such as other Linux systems running Avahi or Macs. I’d installed Avahi earlier and used some service definitions from this article and this blog post to advertise Samba and HTTP. In addition, after installing Firefly, I’d noticed that Firefly starting advertising its presence automatically through Avahi with no service definition required.

Upon installing Netatalk, I also noticed that Netatalk started advertising automatically via Avahi as well, but using the IP address of the server. In order to be able to control how Netatalk advertises via Avahi, I had to change this line in /etc/avahi/avahi-daemon.conf:

enable-dbus=no

The suggestion for this change came from this thread on the Ubuntu Forums. Upon making the change and restarting Avahi, the odd Netatalk entry went away, but so did Firefly! To advertise both Netatalk and Firefly, I added a couple of files to /etc/avahi/services:

afpd.service:

<?xml version="1.0" standalone='no'?><!--*-nxml-*-->
<!DOCTYPE service-group SYSTEM “avahi-service.dtd”>
<service-group>
<name replace-wildcards=”yes”>Intrepid Time Machine</name>
<service>
<type>_afpovertcp._tcp</type>
<port>548</port>
</service>
<service>
<type>_device-info._tcp</type>
<port>0</port>
<txt-record>model=AirPort</txt-record>
</service>
</service-group>

daapd.service:

<?xml version="1.0" standalone='no'?><!--*-nxml-*-->
<!DOCTYPE service-group SYSTEM “avahi-service.dtd”>
<service-group>
<name replace-wildcards=”yes”>Home Music Server</name>
<service>
<type>_daap._tcp</type>
<port>3689</port>
</service>
</service-group>

After placing these two files into /etc/avahi/services, the new services starting advertising immediately. By the way, you’ll note the extra “device-info” entry in afpd.service; that sets the icon that will be used by Macs when they discover this service. I made mine look like a Time Capsule by using the setting listed above.

During this work with Avahi, I uncovered a couple of interesting things:

• I found that restarting the Avahi daemon is actually more problematic than just leaving it alone; in order to make it start advertising services again after a restart, you’ll have to open one of the files in /etc/avahi/services and then close it again. No changes are necessary to the file, but opening it will kickstart Avahi into service advertisement.
• Advertising SMB/CIFS and AFP together with the same name caused my Mac to ignore the SMB/CIFS services and only use AFP. I had to separate SMB/CIFS and AFP into different entries. Since I was using AFP really only for Time Machine backups and SMB/CIFS for everything else, it wasn’t really a big deal.
• Advertising SMB/CIFS and RFB (Screen Sharing, as outlined here) works fine together.

At this point, task #5 was pretty much complete. I had originally envisioned providing file sharing services to the same locations via both AFP and SMB/CIFS, but in the end—partially because of the odd issue with AFP and SMB/CIFS being advertised together as described above—settled for using AFP only for Time Machine and SMB/CIFS for everything else.

Along the way, I also configured screen sharing as outlined here, and it seems to work just fine. I have to leave an account logged in to the Ubuntu server, but I can just lock the screen when I’m not logged in remotely.

The last step was to enable Time Machine backups to the Ubuntu server via AFP. First, the hack to enable non-Time Capsule network backups (this should be all on one line):

defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1

I was then able to select the Ubuntu-hosted AFP volume for Time Machine backups. Attempting to run a Time Machine, backup, however, reported an error about being “unable to create the disk image”. Fortunately, a number of different articles pointed to the use of hdiutil to create the disk image, and that seemed to fix the problem. Time Machine is now backing up to the AFP volume, although I suspect I still have a few issues to work through (for example, it looks as though I have to keep the Time Machine AFP volume mounted in order for automatic backups to run).

So, when everything is said and done, I was able to achieve all my stated goals. The only outstanding issue that I haven’t yet figured out yet centers on automatic logins; for both AFP and SMB/CIFS, I get prompted for a password when connecting, even though I keep my password synchronized (manually) between my Mac and the Ubuntu server. Any tips on how to resolve that would certainly be appreciated.

Along the way, I found the following sites to be quite helpful. I’m sure there are others that I used along the way, and I apologize if I’ve failed to extend credit where credit is due.

Limit size of Time Machine backups on Time Capsule
Set up Time Machine on a NAS in three easy steps
Make Ubuntu a Perfect Mac File Server and Time Machine Volume
Five Guides on How to Integrate Ubuntu into a Mac OS X Network
Time Machine Wireless Backups without Time Capsule

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

Ubuntu and Mac OS X Integration

• Strange Error with Firefly Media Server, iTunes, and Avahi
• My Ubuntu-Mac Article is Popular, I Guess
• Greater AD Integration via NFS and Automounts
• Snow Leopard, Time Machine, and Iomega ix4-200d
• OpenVPN and mt-daapd

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

No Solaris-AD Integration Update

• Solaris-AD Integration Update Coming
• Samba in Solaris-AD Integration
• Solaris 10-AD Integration, Version 3
• Some Things I’m Working On
• Learning Solaris

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

OpenSSH with Kerberos and Active Directory

• SSHjail in Centralized Environments
• Samba and Windows Server 2008 Interoperability
• ESX Server Integration with Active Directory
• New Book on Linux-Windows Integration
• Some Notes on Solaris-AD Integration

It’s certainly very possible to do this with command-line tools such as AdFind or DsQuery, but users may also find it useful to have a saved query available within the Active Directory Users & Computers console for easy reference.

The way to do this is define a custom query using this string:

(objectCategory=Person)(objectClass=User)(uidNumber=*)

If you add just this text and nothing else in the “Find Custom Search” dialog box (the Advanced tab), then the console will automatically add ampersands and additional parentheses to turn it into a “proper” LDAP query that will show you any account that has a UID number configured. Certainly, additional fields like loginShell or unixHomeDirectory could be added as well, but this query will probably be sufficient for most instances.

I started not to publish this, but figured if I couldn’t remember the exact syntax then someone else might not be able to remember the syntax either. This one is as much for me as it is for others.

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

Finding UNIX-Enabled Accounts in Active Directory MMC

• Finding Recently Created Active Directory Accounts
• AD Integration Tip: Dealing With More Than 1,000 Users
• Complex Queries Against Active Directory
• Finding Duplicate Names in Active Directory
• Semi-Automatic Security Groups

Finally, I would like to share one registry change that we’ve found to be necessary in our AD integration. By default, the MS LDAP server only returns 1,000 results. As a university department with more than 1000 active students, this limitation has caused us some frustration.

This KB article shows how to increase the number of results returned in a query: http://support.microsoft.com/kb/315071

We recently set MaxPageSize to 5,000. I don’t know if this will
introduce additional problems down the road, but at least it lets me fully enumerate all our AD users from a Linux machine with `getent passwd`.

If you have an Active Directory domain with more than 1,000 users in the DN specified in your LDAP configuration, then this is a Registry change you’ll want to investigate. Otherwise, you could find that your UNIX/Linux servers aren’t able to fully enumerate all the users in the domain.

Thanks, Scott!

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

AD Integration Tip: Dealing With More Than 1,000 Users

• Using Samba in Linux-AD Integration
• Local Logins Refused in AD Integration Scenarios
• One Potential Issue in AD Integration Scenarios
• Linux-AD Integration Issue: Persistent Connections
• LDAP Signing in AD Integration Situations

So, that being said, here are the notes:

• SSH “shell” access to NetWare 6.5 server: The SSHD NetWare Loadable Module (NLM) had to be loaded first. Attempts to login failed; the sshd_config file had to be edited and a Novell-specific directive (eDirNameContext) had to be modified in order to add the context where the admin account was stored (in this case, OU=Users.O=Company). After the configuration file was modified and the SSHD NLM unloaded and loaded again (to reflect the changes to the configuration file), logins via SSH were successful. (Note: It appears that NetWare 6.5 does not support the Blowfish-CBC cipher.)
• SFTP access to NetWare 6.5 server: After successful SSH “shell” access (see previous bullet point), SFTP access also worked correctly. Tests using Fugu (a native Mac OS X SFTP application) were successful and without any major events or problems. In fact, SFTP was used to transfer the files necessary for the VNC testing (see next bullet point) to the NetWare server.
• VNC access to NetWare 6.5 console: Using SFTP, a VNC server NLM was copied to the server. After setting the VNC password (using VNCPASS.NLM) and loading the VNC server (VNCSRV.NLM), access to the NetWare server’s GUI via VNC was successful. The VNC client used was Chicken of the VNC, a freeware Mac OS X VNC client. Performance was on par for LAN access to a server.
• Native file access from Mac OS X: As indicated in several online sources, the AFPTCP NLM had to be unloaded and then reloaded with the CLEARTEXT option. Then the SYS volume on the server could be mounted using the Go To Server command. After an initial login, the AFPTCP NLM was unloaded and reloaded without the CLEARTEXT option, and everything continued to work just fine.
• Rconsole access from Mac OS X: Using RconJ, a Java-based port of Rconsole to Mac OS X, Rconsole access was successful. The RCONAG6 NLM had to be loaded first on the server in order for this to work.
• VNC inside SSH tunnel: Creating SSH tunnels (using the –L switch) works in NetWare 6.5 just as it does with Linux or OpenBSD. Using the VNC NLM discussed earlier and an SSH tunnel, the VNC traffic was secured and encrypted across the wire. This worked exactly as expected.
• Native file access from Windows XP: Initial attempts to access the server from a Windows XP system failed (authentication problems). The NDS user object had been created in iManager and a simple password had also been created in iManager as well (necessary before CIFS will work). However, the cifsctxs.cfg file (that specifies contexts) had not been updated with the correct context (OU=Users.O=Company, which is where all user objects are stored). After modifying this file and reloading CIFS, then access from Windows XP still failed (network path not found). Further tests showed that typing the UNC path from the Run command on the Start menu failed, but browsing through My Network Places or typing the UNC path including a share name worked just fine.
• NTP on NetWare 6.5: XNTPD.NLM is an NTP daemon for NetWare, similar in implementation and purpose as NTP on Linux or OpenBSD. Upon editing the NTP.CONF file in SYS:\ETC, XNTPD could be loaded only after TIMESYNC.NLM was unloaded. Even then, XNTPD seemed to unload occasionally and without reason, and the NTPDATE utility had to be used to manually synchronize the time.
• Autoloading specific NLMs on startup: Upon reboot, the VNC, SSH, and Rconsole NLMs weren’t loaded, and so the server was inaccessible except from the console. Using the “rconag6 encrypt” command, a LDRCONAG.NCF file was created with an encrypted Rconsole password. Then, AUTOEXEC.NCF was edited to reference this file (in order to load the Rconsole agent) as well as the SSH and VNC NLMs. This would ensure that the necessary NLMs were loaded every time the server booted.
• Universal passwords: After some difficulty mounting a volume from Mac OS X, setting passwords, and such, the server was rebooted and Universal Passwords were enabled for the Users.Company container. The passwords were then set for various accounts. Following that, native file access from both Mac OS X and Windows XP (with one caveat; see below) worked flawlessly. The caveat for Windows XP native file access is that browsing shares using just the server name in the UNC path does not work; at least one share name must also be included (i.e., \vsninteg does not work, but \vsninteg\sys works just fine). SSH access worked fine after enabling universal passwords. SFTP access worked fine as well, as long as the user logging in had sufficient permissions.

OK, there you go. Here’s hoping it may prove useful to someone. Feel free to correct me, clarify these notes, or just tell me I’m crazy in the comments below.

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

Old NetWare Integration Notes

• Access-Based Enumeration
• Using scponly on ESX Server
• VM Portability, Round 3
• Hanging Around #vmware
• Solaris 10 x86 on Mac OS X

We traded e-mails back and forth for a while, and eventually he found the solution himself:

We work with a locked down version of OSs and in this case a domain policy on the Windows server was preventing the RHEL machines from accessing account info.  The policy was “Domain controller: LDAP server signing requirements” which was set to “Require signature.”  When I changed this setting to “None” it worked great.

This is good information and important to keep in mind; I’ll be sure to incorporate this into the next revision of the Linux-AD integration instructions. (No, I don’t have a timeframe on when that will be!)

In the meantime, if anyone has a workaround for this problem that will allow LDAP to work with signatures enabled or required, I’d love to hear it. Speak up in the comments below!

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

LDAP Signing in AD Integration Situations

• Using Samba in Linux-AD Integration
• CentOS 5 Active Directory Integration Problem
• A Couple Cool Mac Discoveries
• Next Integration Task(s)
• Native Kerberos Authentication with SSH

After much trial and error, I had determined what was not the cause the problem:

• The /etc/pam.d/system-auth file was not at fault; we tried numerous combinations in system-auth and there was no difference in behavior
• The /etc/ldap.conf file was not at fault; we even tried adding a few additional entries (like “bind_policy soft”) to help with issues when LDAP was down and not responding
• A lack of DNS resolution was not the problem; the behavior was the same whether DNS was working or not

Finally, I was able to track down this thread which discusses the behavior of the nss_ldap libraries when the LDAP service is not available across the network. In this specific message in the thread, it is noted that nss_ldap will try to contact LDAP to enumerate group membership, even if LDAP is down. So the problem was with using LDAP for group membership, and a quick edit to /etc/nsswitch.conf to remove LDAP from the group line proved that to be true.

As shown in the message, the only workarounds are:

• Upgrade to v245 of nss_ldap, which allows the use of the “nss_initgroups_ignoreusers” directive; this instructs nss_ldap to not perform group enumeration for the specified users; or
• Remove the “ldap” entry from the group line in /etc/nsswitch.conf.

Unfortunately, ESX Server 3.0.2 and ESX Server 3.5.0 only supply nss_ldap v207-17, which is too early to support that directive. Of course, we can’t really upgrade the library, either, since that’s not supported by VMware. So the only real fix for VMware ESX Server AD integration scenarios is to not use Active Directory for group memberships. User accounts can still be managed using Active Directory—and authentication occurs against Active Directory—but groups and group membership will have to be handled locally.

This issue is applicable to other operating systems besides ESX Server, though, and for those operating systems an upgrade of the nss_ldap library and the use of the “nss_initgroups_ignoreuser” directive in ldap.conf may be all that is needed to fix an issue with local logins being refused when network connectivity is not present.

UPDATE: It appears that local logins will work without network connectivity, even with full Active Directory integration, if you use the Emergency Console. Thanks to Magnus for the update!

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

Local Logins Refused in AD Integration Scenarios

• ESX Server-AD Integration
• One Potential Issue in AD Integration Scenarios
• Enumerating Universal Group Membership
• Some Things I’m Working On
• Linux-AD Integration Issue: Persistent Connections

Be aware that Data ONTAP’s multi-mode VIFs are only compatible with static 802.3ad link aggregation; you can’t use PAgP (Cisco proprietary protocol). I would assume dynamic LACP is also incompatible. For this reason we used the “channel-group 1 mode on” statement instead of something like “channel-group 1 mode desirable”.

I recently got some feedback from a NetApp SE in my area; this SE informed me that Link Aggregation Control Protocol (LACP, part of the IEEE 802.3ad specification) is indeed supported with Data ONTAP version 7.2. This KB article on the NetApp NOW site (login required) indicates that ONTAP 7.2.1 is required in order to use a LACP VIF.

There are a couple important requirements to note; these are laid out in the referenced KB article:

1. Dynamic multimode VIFs should use IP address-based load balancing. This means that the Cisco switch or the channel group must also use IP address-based load balancing.
2. Dynamic multimode VIFs must be first-level VIFs. This makes sense; LACP is a Layer 2 protocol, so layering a LACP VIF on top of other VIFs just doesn’t work.

To create the dynamic multimode VIF on the Data ONTAP side, the command is pretty simple:

vif create lacp <vif name> -b ip {interface list}

On the Cisco side, the commands are very similar:

s3(config)#int port-channel1
s3(config-if)#description LACP multimode VIF for netapp1
s3(config-if)#int gi0/23
s3(config-if)#channel-protocol lacp
s3(config-if)#channel-group 1 mode active

These commands would be repeated for all physical ports that should be included in the LACP bundle. Note the differences from the earlier commands in the previous article; here we use “channel-group 1 mode active” instead of “channel-group 1 mode on“. We also added the “channel-protocol lacp” command.

Together, these commands will establish a LACP-based link aggregate between a NetApp storage system running Data ONTAP version 7.2.1 or higher and a Cisco IOS-based switch.

Thanks to Jeff, our NetApp SE, for providing the updated information.

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

LACP with Cisco Switches and NetApp VIFs

• Cisco Link Aggregation and NetApp VIFs
• NetApp VIF Member Limitations
• More on VMware ESX NIC Utilization
• Connecting Nexus 5000 to Older Gigabit Ethernet Switches
• ESX Server, NIC Teaming, and VLAN Trunking