Linux

A while back, I experimented with a multicast DNS (mDNS) responder for Linux.  (For those not already “in the know,” so to speak, multicast DNS is one of the key components of Bonjour, Apple’s automatic service discovery functionality—formerly known as Rendezvous).  For some strange reason, I had an urge to try it again today.  Here’s what I found.

First, I started looking for an “official” RPM package for a CentOS 4.2-based server that I manage.  Despite numerous Google hits that implied an official RPM existed, I could not find one.  (Pointers and/or URLs are welcome.)  I finally found a few RPMs on one of the CentOS mirrors, and installed it without any major issues.  The problem was, there was no documentation.  It installed an executable file called mdnsd, along with a directory in /usr/share/doc and a matching init script.  But how to configure it?  How to tell it what services to advertise via mDNS?

Having no luck whatsoever finding any additional documentation, I turned to a POSIX-compliant mDNS responder I had downloaded from Apple’s developer site and compiled on Red Hat Linux 9.0 some time ago.  I also had a simple init script for it, which (if I recall correctly) had been created by Rui Carmo of Tao of Mac (great site, by the way—I recommend it).  Fortunately for me, all I had to do was just copy the files over to the CentOS-based server and place the files in the right place, and it worked flawlessly.

Sure enough, I could now see this Linux-based server in Terminal.app’s “Connect to Server” dialog box.  I could not, however, see the server as an SFTP server in Cyberduck.  I briefly searched to see what kind of advertisements Cyberduck was expecting to see, but couldn’t find any information.  (Note, strangely enough, that Terminal.app could see the server as an SFTP server, but Cyberduck couldn’t.)

Now don’t ask me why exactly I was driven to tinker with this today, because I couldn’t tell you.

More information on multicast DNS, DNS Service Discovery, and related technologies can be found at the sites linked below:

DNS Service Discovery (DNS-SD) - http://www.dns-sd.org/
Multicast DNS - http://www.multicastdns.org/

Earlier today I completed another GSX Server upgrade (from version 2.5 to version 3.2.1) for a customer, and fortunately this upgrade was much smoother than the last GSX Server upgrade.

No BSODs (Blue Screens of Death) this time during the uninstallation, and the previous version uninstalled itself cleanly.  The installation of the new version went quickly and smoothly, and in practically no time we were booting up “legacy” VMs under the new version of GSX Server.

As with the previous upgrade, those VMs running Red Hat Linux 9.0 detected “new” hardware (specifically, a “new” network card and a “new” SCSI card; ironically, they are exactly the same virtual hardware as before) and seamlessly migrated over the configuration without any issues whatsoever.

Unfortunately, I was dismayed to find that a VM running CentOS 4.2 did not maintain the network configuration while setting up this “new” hardware, so I had to go back in and re-enter the network settings.  This was only a minor inconvenience, as the network reconfiguration was quick and easy.  It’s also nice to note that the time synchronization problems with CentOS 4.2 appear to have resolved themselves now under the new version of GSX Server.  (Note that even after the changes that mostly resolved the NTPd problems, time was still slightly off and lots of NTPd messages were being logged; even those issues have disappeared as well.)

The network reconfiguration on a VM running Windows Server 2003, on the other hand, was not quick and easy.  As before, the network configuration simply disappeared during the discovery of “new” hardware.  In the last upgrade for this customer, the only Windows-based VM we had to work with was an older Windows 2000-based server.  I had hoped that the problems we’d seen with that server would be resolved in Windows Server 2003.

Not so.  The VMware Tools installation removed the driver and installed a new driver, so even if the network configuration had made it through the “new hardware” discovery process, it would have been hosed at that point.  Eventually, after a couple of different reboots, I finally had the Windows server up and running with its original network configuration again.  If there is one area I’ve found so far that VMware really needs to work on, it’s this one.

I had a situation today where a customer forgot the root password to a Debian GNU/Linux 3.1 system in their office.  That left it up to me to try to find a way to get into the system.  Here’s how I managed to gain access.

(Note: As far as I am aware, NONE of the information I’m going to list in this article will work across the network; you MUST have physical access to the server.  Therefore, I’m not too terribly worried about “making it easier for the hackers”.  If you don’t have physical security, then no amount of electronic security is going to help you!)

Here’s how it works:

1. With physical console access, reboot the server.
2. When the Grub menu comes up, press “e” to edit the menu selections.
3. Use the arrow keys to select the Kernel line, then press “e” again.
4. Add “single init=/bin/bash” to the end of the existing line.
5. Press “b” to boot the modified line.
6. The system will boot up into single-user mode.  Unfortunately, the root filesystem will be mounted read-only, so you’ll need to remount it using “mount -o remount,rw /”.
7. Use the “passwd” command to change the password for root to whatever you like.
8. Reboot the computer again and log in as root with the new password.

There are ways to protect against even this (a BIOS-based power-on password, or passwords in Grub to prevent casual editing of the boot configuration), and those steps may be necessary depending upon the other aspects of physical security.  If this system is out where people can get to it, then I’d highly recommend taking these additional steps to secure the server.

Please note that I’ve only done this on Debian GNU/Linux 3.1, but I would be reasonably confident that the steps will work elsewhere as well.

Last week, Novell open sourced the AppArmor software under the GPL, hoping to promote adoption and development of the security framework for Linux.  I was alerted to the move by a number of articles on the subject, such as this article from ComputerWorld and this article from eWeek.

AppArmor competes with Security Enhanced Linux, or SELinux, which—due to its complexity—is often turned off completely.  Clearly, Novell hopes to gain some traction by opening up the source code to AppArmor and providing it to anyone to download.  The software is also being included with OpenSUSE and SLES.

If AppArmor is easier to configure and use than SELinux yet provides comparable enhancements to security, I’ll certainly take a much closer look at it for deployment on my Linux servers.

The last few days have been interesting ones for me.  For those of you that didn’t already know, I’m a network/systems engineer, and as such I’m involved in a variety of projects with a variety of customers.  Just the other day, I had the opportunity to perform an upgrade of VMware GSX Server on Windows Server 2003, from version 2.5 to version 3.2.1.  Here are all the gory details.

Regular readers of this weblog know that I’m a huge fan of virtualization, and VMware’s products—as the industry leaders in this technology—are particular favorites.  I was excited, therefore, to get to participate in this upgrade.  I reviewed the VMware documents in advance, noting the need to uninstall the previous version before installing the new version, and noting that virtual hardware would have to be upgraded before they would be able to take advantage of all the latest features.  OK, seems fairly straightforward, so we shut down the virtual machines (VMs) and got started.

The uninstall of GSX Server 2.5 is reasonably painless, except for the Blue Screen of Death that assails us halfway through the installation.  We still haven’t determined exactly what caused the BSOD, but fortunately the host server booted back up again and showed no signs of further problems.  Thinking that the GSX Server 2.x uninstall was complete given that the product was no longer listed in Add/Remove Programs, I launched the GSX Server 3 installer only to be greeted with a message stating that the previous version had to be uninstalled first.

Fortunately, another trip to Add/Remove Programs showed a GSX Server entry lingering there, and I was able to finally uninstall the application completely (requiring another reboot in the process).  After the server came back up, I installed GSX Server 3.2.1 quickly and without incident.

Next, I made a backup copy of a small Linux VM running a 2.4.x kernel.  This would be the first VM back up and running under the new version of GSX Server.  When the backup was complete, I tried to upgrade the virtual hardware.  The attempt failed; I learned I had to install the latest version of the VMware Tools in order to upgrade the virtual hardware.  OK, fair enough, but these VMs didn’t have the VMware Tools installed because they are text-only servers (they don’t run the X Window System).  For now, the customer and I agree to leave the existing Linux VM intact and just run it “as is.”

I boot the server with “legacy” hardware, and it boots perfectly and the hardware detection application launches, finding some changes in the “hardware” and wanting to reconfigure the hardware.  A bit wary, I allow the application to update the virtual server’s configuration, and I find that the process completes quickly and without any problems.  Note that this reconfiguration included the network card, which had me worried that I’d have to recreate the network configuration.  I did not—that’s good news.

A second text-only Linux VM, also with a 2.4.x kernel, comes up as well and I am pleased to note that it behaves exactly as the first one did.  Repeatable results are good, in my opinion.  Network connectivity to both VMs is fine; their network services are functioning perfectly despite the fact that GSX Server 3 sees them as “legacy VMs”.  Now on to a Windows-based VM.

I pick a non-essential Windows VM for testing.  The initial boot is identical to the others.  Once I get logged in, I install the new VMware Tools, which installs new versions of the video, mouse, network, audio, and SCSI drivers.  Unfortunately, during the process of installing the updated drivers, the previous versions are completely removed.  This completely erases the network configuration of the Windows-based virtual server, which I must then recreate.

<aside>This is very irritating.  VMware needs to work on this process.  In this case, it wasn’t a huge deal, but with a large number of VMs that would have to be reconfigured it could get ugly real quick.  It can be done with Linux, why not with Windows?  OK, that’s probably a loaded question.</aside>

Once the network settings are properly reconfigured, I shut the server down and upgraded the virtual hardware.  This process took far longer than I had anticipated it would, apparently needing to somehow reconfigure the virtual disk files.  After a lengthy delay, the virtual hardware upgrade is complete and I boot the Windows VM again.  As expected, the “New Hardware Wizard” launches once I get logged into the VM.  Not as expected, however, the wizard can’t find drivers for the virtual audio device that has been introduced as a result of the virtual hardware upgrade.  I disable the device and continue; servers don’t need audio anyway.

I still have a couple more virtual machines to boot as of this writing; however, these are pre-production systems (still in testing and validation) so there is no terrible rush just yet.  So far, I’m both pleased and displeased with GSX Server 3.2.1 compared to previous versions.  I have another server to upgrade soon; that upgrade will be particularly interesting because it hosts some Linux VMs with a 2.6.x kernel.  These 2.6.x kernels seem to be a bit problematic under GSX Server 2.5.

As the upgrades continue, I’ll post additional information.  In the meantime, if you are a VMware expert and want to post some comments telling me how I could have handled this situation better, please do!  I’m always seeking information to help me improve.

As I’ve mentioned in a couple of previous posts, I have several servers running Red Hat Linux 9.0 that I am looking to upgrade to a newer distribution.  I’ve tried a couple of recent versions of CentOS (a clone version of Red Hat Enterprise Linux), but ran into problems with ntpd (I may have finally resolved those).  I’d heard good things about Debian GNU/Linux, so I decided to give that a try as well.

Prior experience with Ubuntu (as well as some time with Kubuntu) led me to believe that Debian would be equally polished.  So, I downloaded two ISO’s for Debian 3.1 (“Sarge”), the latest stable release.  Information I’d gained from some online research indicated by this release included a 2.6.8 version of the kernel, but upon installation I found I was running a patched 2.4 version instead.  After a couple of attempts to upgrade the kernel via apt-get, I finally managed to locate an appropriate package name in the stable distribution and installed it.

Unfortunately, the 2.6.8 kernel cause the system to fail to boot, listing numerous segmentation faults and finally just halting partway into the boot process.  I was extremely disappointed—after having spent that time downloading the software and then installing it, it was now completely unusable.

I’m sure that Debian fans will point out that I surely did something incorrect and things normally don’t happen that way.  Perhaps that is true; I will certainly be the first to admit that I may have done something incorrectly while using apt-get to upgrade the kernel.  I would be nice to know exactly what it was that went wrong, though.

In the meantime, I’m going to shelve Debian GNU/Linux as a possible replacement OS and continue searching.  At this point, OpenBSD is starting to look really good…

The NTPd problem that I wrestled with in CentOS 4.1 and again in CentOS 4.2 has finally been resolved.  Mostly.  I think.  The specific steps I took to resolve the issue came from a number of sources, so read on for all the details.

Since these servers are virtual servers running under VMware, I first consulted the VMware Knowledge Base and turned up this article on slow and fast clocks for Linux guest VMs.  Based on that information, I added a few extra commands to the grub configuration:

noapic nosmp nolapic clock=pit

In addition, I found a number of forum postings in various sites (too many to list or link here) that referenced problems with NTP and ACPI.  So, based on that information, I further edited the grub configuration to look like this:

noapic nosmp nolapic clock=pit acpi=no

Finally, based on information regarding NTP itself and the NTP configuration parameters, I added the “burst iburst” parameters to the server lines in my ntp.conf file, like this:

server W.X.Y.Z burst iburst

This helped, as at least now NTP would synchronize against something other than the local clock (which was more than it had done previously).  For some reason, though, the /var/log/messages log file was filling up with messages about synchronizing against the local clock, then synchronizing against the server, then against the local clock, etc.  (You get the picture.)

Given that I was synchronizing against a Windows Server 2003-based computer, I thought perhaps that Microsoft’s NTP implementation was simply broken.  (This certainly wouldn’t be the first time.)  So I configured OpenNTPd on an OpenBSD server (running OpenBSD 3.8) and re-configured the CentOS server to synchronize against the OpenBSD NTP server.

<aside>OpenNTPd is ridiculously simple to configure and operate, and just works.</aside>

The repetitive synchronization messages still appeared, but were appearing with far less frequency.  And the time still isn’t synchronized to the level I would like, but it does stay within a minute or so of the rest of the network (well within the 5 minute gap required in order for Kerberos authentication to work).

So, it’s still not working as cleanly as the older Red Hat Linux 9.0-based servers, but it is working.  Given that I’m still running these servers under an old version of VMware (which, technically, doesn’t support Linux 2.6 kernels) I may try upgrading to a new version of VMware to see if that helps at all.

UPDATE:  These instructions are for Windows 2000 Server and Windows Server 2003 pre-R2.  For the R2 release, please see these updated instructions.

There are several articles posted here that discuss, in general terms, how to authenticate Linux against Active Directory.  For example, see this brief article on the broad direction of the integration, or this posting on using computer accounts for Linux servers (most how-to documents I’ve seen discuss the use of a user account for this purpose).

Over the last week or so, I’ve been having to rebuild one Linux server a number of times in an effort to fix a separate problem (see this article on NTPd problems with CentOS 4.1 and the recurrence of the problem with CentOS 4.2).  As a result, I’ve had to go through the process of configuring this Linux server (and preparing Active Directory) for authentication several times, and I found that I kept having to go back to my scattered and dispersed notes to remember what had to be done.  In the hopes of correcting that, I am collecting all the pertinent information I have here in this article.

Preparing Active Directory (One-Time)

There is a one-time preparation of Active Directory that is required.  In order to authenticate Linux logins against Active Directory, Active Directory’s schema must be extended to include Linux-specific attributes such as home directory, UID, GID, and default shell.  There are a couple of different ways to do this; I chose to use Microsoft’s Services for Unix (SFU) to extend the schema.  There are two reasons I chose SFU: 1) it’s free; and 2) the SFU schema is being included by default in Windows Server 2003 R2.

Once SFU has been installed and the schema has been extended, then the specific Active Directory accounts that are allowed to authenticate via Linux must be configured with a UID and other UNIX attributes.  This is accomplished via the new “UNIX Attributes” tab on the properties dialog box of a user account.

You’ll also need to create an account in Active Directory that will be used to bind to Active Directory for LDAP queries.  This account does not need any special privileges; in fact, making the account a member of Domain Guests and not a member of Domain Users is perfectly fine.

After all the user accounts have been configured, then we are ready to perform the additional tasks within Active Directory and on the Linux server that will enable the authentication.

Preparing Active Directory (Each Server)

For each server that will be authenticating against Active Directory, follow the steps below.

1. Create a computer account in Active Directory.  When creating the computer account, be sure to specify that this account may be used by a pre-Windows 2000–based computer.
2. Use the following command at a command prompt to configure the new computer account:
   ktpass -princ host/fqdn@REALM -mapuser DOMAIN\name$
-crypto DES-CBC-MD5 -pass password -ptype KRB5_NT_PRINCIPAL
-out filename
   Of course, you’ll need to substitute the appropriate values for “fqdn” (the fully-qualified domain name of the computer), “REALM” (the DNS name of your Active Directory domain in UPPERCASE), “DOMAIN” (the NetBIOS name of your Active Directory domain), “password” (the password that will be set for the new computer account), and “filename” (the keytab that will be generated and must be copied over to the Linux computer).

If you need to rebuild the Linux server for whatever reason, you’ll need to delete the computer account you created and repeat this process.

Preparing the Linux Server

Follow the steps below to configure the Linux server for authentication against Active Directory.

1. Make sure that the appropriate Kerberos libraries, OpenLDAP, pam_krb5, and nss_ldap are installed.  If they are not installed, install them.
2. Be sure that time is being properly synchronized between Active Directory and the Linux server in question.  Kerberos requires time synchronization.
3. Edit the krb5.conf file to look something like this, substituting your actual host names and domain names where appropriate:
   [logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
 
[libdefaults]
default_realm = EXAMPLE.COM
dns_lookup_realm = true
dns_lookup_kdc = true
 
[realms]
EXAMPLE.COM = {
kdc = host.example.com:88
admin_server = host.example.com:749
default_domain = example.com
}
 
[domain_realm]
.example.com = EXAMPLE.COM
example.com = EXAMPLE.COM
 
[kdc]
profile = /var/kerberos/krb5kdc/kdc.conf
 
[appdefaults]
pam = {
debug = false
ticket_lifetime = 36000
renew_lifetime = 36000
forwardable = true
krb4_convert = false
}
4. Edit the /etc/ldap.conf file to look something like this, substituting the appropriate host names, domain names, account names, and distinguished names (DNs) where appropriate.
   host 10.10.10.10
base dc=example,dc=com
binddn cn=ldap,cn=Users,dc=example,dc=com
bindpw adldapbindpw
scope sub
ssl no
nss_base_passwd dc=example,dc=com
nss_base_shadow dc=example,dc=com
nss_base_group dc=example,dc=com
nss_map_objectclass posixAccount user
nss_map_objectclass shadowAccount user
nss_map_objectclass posixGroup group
nss_map_attribute uid sAMAccountName
nss_map_attribute uidNumber msSFU30UidNumber
nss_map_attribute gidNumber msSFU30GidNumber
nss_map_attribute loginShell msSFU30LoginShell
nss_map_attribute gecos name
nss_map_attribute userPassword msSFU30Password
nss_map_attribute homeDirectory msSFU30HomeDirectory
nss_map_attribute uniqueMember msSFU30PosixMember
nss_map_attribute cn cn
5. Securely copy the file created using the ktpass.exe utility above to the Linux server in question, placing it in the /etc directory as krb5.keytab.  (SFTP or SCP are excellent candidates for this.)
6. Configure PAM (this varies according to Linux distributions) to use pam_krb5 for authentication.  Many modern distributions use a stacking mechanism whereby one file can be modified and those changes will applied to all the various PAM-aware services.  For example, in Red Hat-based distributions, the system-auth file is referenced by most other PAM-aware services.
7. Edit the /etc/nsswitch.conf file to include “ldap” as a lookup source for passwd, shadow, and groups.

That should be it.  Once you do that, you should be able to use kinit from a Linux shell prompt (for example, “kinit aduser”) and generate a valid Kerberos ticket for the specified Active Directory account.

At this point, any PAM-aware service that is configured to use the stacked system file (such as the system-auth configuration on Red Hat-based distributions) will use Active Directory for authentication.  Note, however, that unless you also add the pam_mkhomedir.so module in the PAM configuration, home directories will have to be created manually for any Active Directory account that may log on to that server.  (I generally recommend the use of pam_mkhomedir.so in this situation.)

A while back, I started working with CentOS (version 4.1 at the time, see my impressions here) as a candidate replacement for the Red Hat Linux 9.0-based servers still in service on the network.  Although I was (and still am) very comfortable with Red Hat Linux 9.0, it’s getting a bit long in the tooth and I wanted to update the servers to a more modern software base.

The problem is, I kept running into issues with time synchronization on the CentOS server.  This time synchronization issue did not appear on the Red Hat Linux servers.  I tried to resolve the issue, but never did find the solution.  Instead, I just used ntpdate in a cron job.  That seemed to work well enough, but I really wanted it to work PROPERLY.

So I started working with CentOS 4.2 earlier this week, hoping that whatever had plagued me with the previous version wouldn’t afflict me this time.  Unfortunately, the exact same problem has cropped up again, and I am at a loss trying to figure out why it doesn’t work.  The output from “ntpq -p” indicates that communication is occurring with the desired time servers, but the NTP daemon still keeps choosing to synchronize with the local clock.  SELinux is disabled, so it can’t be interfering with NTP (I think that was the problem with the earlier CentOS 4.1 installation), iptables is not running (so it can’t be a firewall issue), file permissions and ownership look OK—what could I be missing? If anyone has any ideas, please let me know.

Before I switched to Mac OS X, I was a heavy duty Windows user.  I had Windows XP on my laptop, Windows 2000 on my servers, and used a Windows CE-based Compaq iPaq H3765.  Since that time, I (of course) switched to Mac OS X on my laptop (a 15“ PowerBook G4 with SuperDrive); a mix of Linux, OpenBSD, and Windows on my servers; and a Palm OS-based handheld (currently a Treo 650, previously a Tungsten T3).  As a result, I still had this iPaq sitting around doing nothing.  So, on a whim, I decided to install Linux on my old iPaq.

Using the installation instructions at familiar.handhelds.org, I downloaded an Opie image, installed the bootloader, and reflashed the iPaq from a spare 128MB CF card (I had a CF sleeve with external battery for the iPaq).  After only a few minutes (the entire process was really quite painless), I was up and running with Linux on my iPaq.

Having Linux running on the iPaq is great, but now I have to figure out what to do with this Linux-based PDA.  Suggestions, anyone?