blog.scottlowe.org

A few weeks ago I wrote about trying to use NetApp’s ONTAP Simulator as a VM under ESX Server so that I could do some testing with the new NAS and iSCSI functionality in ESX Server 3.0.  I finally got that working, but later had to shut it down when I started working with ESX Server 3.  As it turns out, the ProLiant 6400R servers I was using for my VMware lab (running ESX 2.5.x) were not supported for the final release of ESX 3 because the cpqarray.o driver was dropped from the final release, and the cpqarray.o driver is what supported the Smart Array 3200 and Smart Array 4250 RAID controllers in these two boxes.  No ESX 3 on these boxes.

Thankfully, I was able to locate an unused ProLiant ML350 G4p on which I could run ESX Server 3.  Instead of trying to use the ONTAP Simulator as a VM, then, I rebuilt one of the 6400R servers with Red Hat Linux 9.0 and installed the NetApp ONTAP Simulator directly from there.  After a fair amount of work, I finally had everything setup, and the simulated Filer was running with about 20GB of available storage to serve up via iSCSI, CIFS, NFS, or HTTP.

iSCSI was what I was really interested in, so the available storage from the ONTAP Simulator got carved into a couple of LUNS to be presented via iSCSI, along with the requisite initiator groups.  Once everything looked to be in place on the Simulator, I moved to the configuration of ESX Server.  Had I been lucky, I might have been trying to do this after Mike Laverick released his Service Console Guide, but I was a few days too early, and used the VI Client GUI instead to configure the VM Kernel NIC and the software iSCSI client.  The configuration looked correct, but there was no connectivity.

Not sure if the problem was ESX Server or the ONTAP Simulator, I mapped another LUN and created another initiator group and tested iSCSI connectivity from a Windows Server 2003 VM using Microsoft’s iSCSI initiator.  That worked perfectly—not even the first problem.  Clearly, the problem was not with the ONTAP Simulator, but with ESX Server instead.

Reviewing the configuration, I did find a couple of problems:

• The iSCSI node name I had specified in the igroup configuration on the Simulator was incorrect, so I fixed that.  (Or thought I had; more on that in a moment.)
• The iSCSI security configuration was incorrect, so I fixed that as well.

ESX Server should see the storage now, but there was still no connectivity.  Finally, I came across a blurb somewhere about the new firewall in ESX Server 3.0, and how it controlled not only inbound traffic, but also outbound traffic.  A quick trip to the service console and this command:

esxcfg-firewall -e swISCSIClient

You would expect that using the VI Client to enable the software iSCSI client would also enable outbound iSCSI traffic support on the ESX firewall, but it didn’t.  As soon as I entered that command, the Simulator’s console (where I was logged in) started showing iSCSI connection requests.  This, in turn, revealed another problem—ESX Server insisted on using its own iSCSI node name instead of the node name I had assigned.  That was easily and quickly corrected, and I was finally able to mount the iSCSI LUN and create a VMFS datastore.

Key points to remember:

• Apparently, the iSCSI node name you specify in configuring ESX Server will be ignored, so don’t bother.  Just use whatever ESX is already configured with.
• Be sure to either configure iSCSI from the command line (where outbound iSCSI traffic is allowed through the firewall automatically) or go back and allow outbound iSCSI traffic through the ESX firewall.

Having iSCSI-based storage eliminates one potential block to testing and demonstrating VMotion to customers—shared storage—but now I need to work on getting Gigabit Ethernet into the test lab as well.  Too bad there’s not a software workaround for that, too…

UPDATE:  I corrected the command-line above to reflect that the first “I” should be uppercase, not lowercase as was previously noted.  Thanks to Chauncey for catching the error!

In a recent article discussing Novell’s leadership change, one analyst was quoted regarding the change as being positive for Novell in that they (Novell) could stop building very high quality products and instead build products that are just good enough.  I don’t know about you, but this spirit of mediocrity is exactly the wrong kind of thinking for IT vendors.

Specifically, the quote stated this:

“Ron Hovsepian appears to be an astute business person, one who will be able to quickly take stock of the environment and Novell’s position within that environment. This, I hope, will help Novell move from its current position of very slowly building extremely high quality products to quickly building and marketing products that are good enough to satisfy the market,” concluded Kusnetzky.

So what is he (Dan Kusnetzky) proposing then?  It sounds to me that Dan thinks IT software vendors should make their products just good enough to pass muster, instead of making them the best that they can be.

In my opinion, this spirit of mediocrity—this willingness to accept products that are knowingly released with imperfections and flaws because they are “good enough”—is exactly what brought the industry to where it is today.  This mediocrity is what brought SQL Slammer, Blaster, and Melissa.  This is the view that accepts that rebooting your computer a few times a day is just a part of life, and that our operating systems and applications shouldn’t be expected to be stable and reliable.  Just good enough?  When was the last time you recommended a product, service, or vendor because they were “just good enough”?  No, just good enough isn’t good enough.

Every major IT vendor out there—from HP, IBM, and Sun, to Apple, Microsoft, and Red Hat—should be held accountable for the quality of the products they release.  Hey, I understand that companies may make mistakes, and miss errors.  That’s understandable.  But any company that knowingly releases a product that’s “just good enough” when it could have been better is not a company we should be praising.  We should be supporting those companies that emphasize quality over “just good enough”.

Perhaps I’m overreacting.  Perhaps the analyst’s comments were merely directed at the speed with which Novell releases their products, and was instead trying to state that Novell needed to release competing products more quickly.  Even so, any vendor that values speed over quality is bound to get bitten sooner or later.  Microsoft got bitten, and changed their priorities (somewhat).  Apple will get bitten, too, if they start letting the quality of Mac OS X releases slide in favor of shorter development cycles.  The same goes for all the other vendors.

What about you?  I’d love to hear your comments on the matter.

UPDATE:  An updated version of these instructions has been posted.

The integration of (what was formerly called) Services for UNIX into Windows Server 2003 R2 also brought some other changes.  To accommodate those changes, I’ve updated my Linux-AD integration instructions (the previous instructions are here for pre-R2 versions of Windows).  If you need to integrate Linux systems for authentication into Active Directory with Windows Server 2003 R2, these instructions should get you there.

Overall, the instructions are very similar to the instructions for pre-R2 versions of Windows.

Preparing Active Directory (One-Time)

Based on what I’ve seen so far, it appears as if a partial RFC 2307-compliant schema is included by default with Windows Server 2003 R2.  This means that it is no longer necessary to extend the schema to include attributes such as uid, gid, login shell, etc.  However, while the schema does appear to be present by default, you must install the “Server for NIS” component on at least one domain controller in order to be able to actually set those attributes (and it will be necessary to set the attributes before logins from Linux will work).

You’ll also need to create an account in Active Directory that will be used to bind to Active Directory for LDAP queries.  This account does not need any special privileges; in fact, making the account a member of Domain Guests and not a member of Domain Users is perfectly fine.

Preparing Active Directory (Each User)

Each Active Directory account that will authenticate via Linux must be configured with a UID and other UNIX attributes.  This is accomplished via the new “UNIX Attributes” tab on the properties dialog box of a user account.  Installing the “Server for NIS” component enables this, as mentioned previously.

After all the user accounts have been configured, then we are ready to perform the additional tasks within Active Directory and on the Linux server that will enable the authentication.

Preparing Active Directory (Each Server)

For each Linux-based server that will be authenticating against Active Directory, follow the steps below.

1. Create a computer account in Active Directory.  When creating the computer account, be sure to specify that this account may be used by a pre-Windows 2000–based computer.
2. Use the following command at a command prompt to configure the new computer account:

   ktpass -princ host/fqdn@REALM -mapuser DOMAINname$
   -crypto DES-CBC-MD5 -pass password -ptype KRB5_NT_PRINCIPAL
   -out filename

   Of course, you’ll need to substitute the appropriate values for “fqdn” (the fully-qualified domain name of the computer), “REALM” (the DNS name of your Active Directory domain in UPPERCASE), “DOMAIN” (the NetBIOS name of your Active Directory domain), “password” (the password that will be set for the new computer account), and “filename” (the keytab that will be generated and must be copied over to the Linux computer).

If you need to rebuild the Linux server for whatever reason, you’ll need to delete the computer account you created and repeat this process.

Preparing Each Linux Server

Follow the steps below to configure the Linux server for authentication against Active Directory.

1. Make sure that the appropriate Kerberos libraries, OpenLDAP, pam_krb5, and nss_ldap are installed.  If they are not installed, install them.
2. Be sure that time is being properly synchronized between Active Directory and the Linux server in question.  Kerberos requires time synchronization.
3. Edit the krb5.conf file to look something like this, substituting your actual host names and domain names where appropriate:

   [logging]
    default = FILE:/var/log/krb5libs.log
    kdc = FILE:/var/log/krb5kdc.log
    admin_server = FILE:/var/log/kadmind.log
   
   [libdefaults]
    default_realm = EXAMPLE.COM
    dns_lookup_realm = true
    dns_lookup_kdc = true
   
   [realms]
    EXAMPLE.COM = {
     kdc = host.example.com:88
     admin_server = host.example.com:749
     default_domain = example.com
    }
   
   [domain_realm]
    .example.com = EXAMPLE.COM
    example.com = EXAMPLE.COM
   
   [kdc]
    profile = /var/kerberos/krb5kdc/kdc.conf
   
   [appdefaults]
    pam = {
      debug = false
      ticket_lifetime = 36000
      renew_lifetime = 36000
      forwardable = true
      krb4_convert = false
    }

4. Edit the /etc/ldap.conf file to look something like this, substituting the appropriate host names, domain names, account names, and distinguished names (DNs) where appropriate.

   host 10.10.10.10
   base dc=example,dc=com
   uri ldap://server.example.com/
   binddn ldap@example.com
   bindpw adldapbindpw
   scope sub
   ssl no
   nss_base_passwd dc=example,dc=com?sub
   nss_base_shadow dc=example,dc=com?sub
   nss_base_group dc=example,dc=com?sub
   nss_map_objectclass posixAccount user
   nss_map_objectclass shadowAccount user
   nss_map_objectclass posixGroup group
   nss_map_attribute gecos name
   nss_map_attribute homeDirectory unixHomeDirectory
   nss_map_attribute uniqueMember member

5. Securely copy the file created using the ktpass.exe utility above to the Linux server in question, placing it in the /etc directory as krb5.keytab.  (SFTP or SCP are excellent candidates for this.)
6. Configure PAM (this varies according to Linux distributions) to use pam_krb5 for authentication.  Many modern distributions use a stacking mechanism whereby one file can be modified and those changes will applied to all the various PAM-aware services.  For example, in Red Hat-based distributions, the system-auth file is referenced by most other PAM-aware services.
7. Edit the /etc/nsswitch.conf file to include “ldap” as a lookup source for passwd, shadow, and groups.

That should be it.  Once you do that, you should be able to use kinit from a Linux shell prompt (for example, “kinit aduser”) and generate a valid Kerberos ticket for the specified Active Directory account.

At this point, any PAM-aware service that is configured to use the stacked system file (such as the system-auth configuration on Red Hat-based distributions) will use Active Directory for authentication.  Note, however, that unless you also add the pam_mkhomedir.so module in the PAM configuration, home directories will have to be created manually for any Active Directory account that may log on to that server.  (I generally recommend the use of pam_mkhomedir.so in this situation.)

This configuration was tested on Red Hat Linux 9.0 as well as CentOS 4.3.

There is an incredible amount of momentum growing around the open source Xen hypervisor, and it is increasingly looking like market leaders VMware and Microsoft should be less worried about each other and more worried about Xen.

Check out some of the recent news articles regarding Xen:

“Novell to integrate Xen 3.0 in the next Open Enterprise Server”
<http://searchopensource.techtarget.com/columnItem/0,294698,sid39_gci1176440,00.html>

“Virtual Iron annonces 3.0 commercial and free editions based on Xen”
<http://www.virtualiron.com/news_events/releaseDate-4-03-06.cfm>

“Red Hat Formally Announces ‘Integrated Virtualization’”
<http://www.redhat.com/about/news/prarchive/2006/virtualization.html>

“Virtual Iron, XenSource to Unveil Xen 3.0 Products at LinuxWorld”
<http://www.eweek.com/article2/0,1759,1945398,00.asp>

(Note:  all links are courtesy of virtualization.info.)

And those links are just from the last few days!  Clearly, there is lots of momentum and lots of support from big name vendors such as HP, IBM, Novell, Red Hat, and others around the Xen open source hypervisor.  While some have speculated that VMware’s move to release VMware Server for free (and Microsoft’s corresponding drop in price for Virtual Server 2005 R2) have been to stave off each other, perhaps their moves were in response to Xen instead?

A while back, I experimented with a multicast DNS (mDNS) responder for Linux.  (For those not already “in the know,” so to speak, multicast DNS is one of the key components of Bonjour, Apple’s automatic service discovery functionality—formerly known as Rendezvous).  For some strange reason, I had an urge to try it again today.  Here’s what I found.

First, I started looking for an “official” RPM package for a CentOS 4.2-based server that I manage.  Despite numerous Google hits that implied an official RPM existed, I could not find one.  (Pointers and/or URLs are welcome.)  I finally found a few RPMs on one of the CentOS mirrors, and installed it without any major issues.  The problem was, there was no documentation.  It installed an executable file called mdnsd, along with a directory in /usr/share/doc and a matching init script.  But how to configure it?  How to tell it what services to advertise via mDNS?

Having no luck whatsoever finding any additional documentation, I turned to a POSIX-compliant mDNS responder I had downloaded from Apple’s developer site and compiled on Red Hat Linux 9.0 some time ago.  I also had a simple init script for it, which (if I recall correctly) had been created by Rui Carmo of Tao of Mac (great site, by the way—I recommend it).  Fortunately for me, all I had to do was just copy the files over to the CentOS-based server and place the files in the right place, and it worked flawlessly.

Sure enough, I could now see this Linux-based server in Terminal.app’s “Connect to Server” dialog box.  I could not, however, see the server as an SFTP server in Cyberduck.  I briefly searched to see what kind of advertisements Cyberduck was expecting to see, but couldn’t find any information.  (Note, strangely enough, that Terminal.app could see the server as an SFTP server, but Cyberduck couldn’t.)

Now don’t ask me why exactly I was driven to tinker with this today, because I couldn’t tell you.

More information on multicast DNS, DNS Service Discovery, and related technologies can be found at the sites linked below:

DNS Service Discovery (DNS-SD) - http://www.dns-sd.org/
Multicast DNS - http://www.multicastdns.org/

Earlier today I completed another GSX Server upgrade (from version 2.5 to version 3.2.1) for a customer, and fortunately this upgrade was much smoother than the last GSX Server upgrade.

No BSODs (Blue Screens of Death) this time during the uninstallation, and the previous version uninstalled itself cleanly.  The installation of the new version went quickly and smoothly, and in practically no time we were booting up “legacy” VMs under the new version of GSX Server.

As with the previous upgrade, those VMs running Red Hat Linux 9.0 detected “new” hardware (specifically, a “new” network card and a “new” SCSI card; ironically, they are exactly the same virtual hardware as before) and seamlessly migrated over the configuration without any issues whatsoever.

Unfortunately, I was dismayed to find that a VM running CentOS 4.2 did not maintain the network configuration while setting up this “new” hardware, so I had to go back in and re-enter the network settings.  This was only a minor inconvenience, as the network reconfiguration was quick and easy.  It’s also nice to note that the time synchronization problems with CentOS 4.2 appear to have resolved themselves now under the new version of GSX Server.  (Note that even after the changes that mostly resolved the NTPd problems, time was still slightly off and lots of NTPd messages were being logged; even those issues have disappeared as well.)

The network reconfiguration on a VM running Windows Server 2003, on the other hand, was not quick and easy.  As before, the network configuration simply disappeared during the discovery of “new” hardware.  In the last upgrade for this customer, the only Windows-based VM we had to work with was an older Windows 2000-based server.  I had hoped that the problems we’d seen with that server would be resolved in Windows Server 2003.

Not so.  The VMware Tools installation removed the driver and installed a new driver, so even if the network configuration had made it through the “new hardware” discovery process, it would have been hosed at that point.  Eventually, after a couple of different reboots, I finally had the Windows server up and running with its original network configuration again.  If there is one area I’ve found so far that VMware really needs to work on, it’s this one.

As I’ve mentioned in a couple of previous posts, I have several servers running Red Hat Linux 9.0 that I am looking to upgrade to a newer distribution.  I’ve tried a couple of recent versions of CentOS (a clone version of Red Hat Enterprise Linux), but ran into problems with ntpd (I may have finally resolved those).  I’d heard good things about Debian GNU/Linux, so I decided to give that a try as well.

Prior experience with Ubuntu (as well as some time with Kubuntu) led me to believe that Debian would be equally polished.  So, I downloaded two ISO’s for Debian 3.1 (“Sarge”), the latest stable release.  Information I’d gained from some online research indicated by this release included a 2.6.8 version of the kernel, but upon installation I found I was running a patched 2.4 version instead.  After a couple of attempts to upgrade the kernel via apt-get, I finally managed to locate an appropriate package name in the stable distribution and installed it.

Unfortunately, the 2.6.8 kernel cause the system to fail to boot, listing numerous segmentation faults and finally just halting partway into the boot process.  I was extremely disappointed—after having spent that time downloading the software and then installing it, it was now completely unusable.

I’m sure that Debian fans will point out that I surely did something incorrect and things normally don’t happen that way.  Perhaps that is true; I will certainly be the first to admit that I may have done something incorrectly while using apt-get to upgrade the kernel.  I would be nice to know exactly what it was that went wrong, though.

In the meantime, I’m going to shelve Debian GNU/Linux as a possible replacement OS and continue searching.  At this point, OpenBSD is starting to look really good…

A while back, I started working with CentOS (version 4.1 at the time, see my impressions here) as a candidate replacement for the Red Hat Linux 9.0-based servers still in service on the network.  Although I was (and still am) very comfortable with Red Hat Linux 9.0, it’s getting a bit long in the tooth and I wanted to update the servers to a more modern software base.

The problem is, I kept running into issues with time synchronization on the CentOS server.  This time synchronization issue did not appear on the Red Hat Linux servers.  I tried to resolve the issue, but never did find the solution.  Instead, I just used ntpdate in a cron job.  That seemed to work well enough, but I really wanted it to work PROPERLY.

So I started working with CentOS 4.2 earlier this week, hoping that whatever had plagued me with the previous version wouldn’t afflict me this time.  Unfortunately, the exact same problem has cropped up again, and I am at a loss trying to figure out why it doesn’t work.  The output from “ntpq -p” indicates that communication is occurring with the desired time servers, but the NTP daemon still keeps choosing to synchronize with the local clock.  SELinux is disabled, so it can’t be interfering with NTP (I think that was the problem with the earlier CentOS 4.1 installation), iptables is not running (so it can’t be a firewall issue), file permissions and ownership look OK—what could I be missing? If anyone has any ideas, please let me know.