CentOS

You are currently browsing articles tagged CentOS.

In early January 2012, I posted a list of my 2012 projects. Those projects included learning to script in Perl, learning German, becoming more familiar with Xen/Open vSwitch/OpenStack, and pursuing CCNP. In late June 2012, my mid-year project update let you know that I was dropping the Perl efforts, but sticking it out with German, Xen/OVS/OpenStack, and CCNP. (I later dropped Xen in favor of KVM.) Finally, in early January 2013 I graded myself on my progress on my 2012 goals. In this post, I’d like to take lessons learned from my 2012 projects and apply them toward my list of 2013 projects.

As a quick summary, here are some key “lessons learned” from my 2012 projects:

  • The synergy of the projects does make a difference. I should try to find projects that are related and that support one another in some fashion.
  • Projects need to be more tightly defined. Projects that are overly broad will have mixed results.
  • There are still some fundamental building blocks that I need to reinforce.

With the conclusions listed above in mind, here is my list of 2013 projects and—in some cases—some proposed goals for which I’ll be aiming.

  1. Continue to learn German. Obviously, this is a continuation of my 2012 project to learn to speak German. This year, I’ll need to re-evaluate ways to enhance my motivation, and find additional ways to reinforce the information I’m learning. I don’t have any specific goals in mind for this project, although that is something I’ll be evaluating over the course of the year (I do agree that clear goals—something I didn’t establish last year—can help with progress.)

  2. Reinforce base Linux knowledge. This is one of the “fundamental building blocks” that I needed to reinforce. I’ll be focusing on Red Hat Enterprise Linux (RHEL) and RHEL variants, since that seems to be what’s most commonly found in enterprise deployments today. My primary goal here is to be able to do most anything that I need to do without constantly having to look up a “how to” guide; a secondary goal that I’m considering (but haven’t decided upon) would be to look at a Red Hat-focused certification (like RHCE).

  3. Continue using Puppet for automation. This will be partly tied to #2. I’ve mentioned elsewhere that you can’t automate something you don’t fully understand, so as my base Linux knowledge is reinforced I will also be seeking ways to automate tasks with Puppet. As with my base Linux knowledge, my primary goal here is to be able to do most anything that I need to do with Puppet. I might consider Puppet certification, but I think that’s a long shot.

  4. Expand and deepen data center networking fundamentals. Making progress on CCNP was one of my 2012 goals, but it didn’t really fit in with the rest of my focus areas (which were almost all centered in the data center). To help improve the synergy of projects, then, I’ve decided that my networking focus should be DC-oriented. This also ties in nicely with my new job role at Nicira/VMware. My primary goal here is simply to be more knowledgeable, although over the course of the year I will consider pursuing CCNA-DC.

So, that’s my list of 2013 projects. I’m sure that I’ll have a few side projects here and there as well (for example, there is a book project and at least one video training series project on the books for 2013), but the projects listed above represent the majority of my technical focus over the coming year. I’d love to hear your feedback—am I missing something important? Are there ways I could further improve the synergy of the projects? As always, your honest and courteous comments are welcome.

Tags: , , , , ,

In this third post on using Mock to build RPMs for CentOS 6.3, I’m going to show you how to use Mock to build RPMs for Libvirt 1.0.1 that you can install on CentOS. As you’ll see later, this post builds on the previous two posts (one on using Mock to build RPMs for sanlock 2.4 and one on using Mock to build RPMs for libssh2 1.4.1).

Here’s a quick overview of the process:

  1. Set up Mock and the environment.
  2. Install prerequisites into the Mock environment.
  3. Build the Libvirt RPMs.

Let’s take a closer look at each of these steps.

Setting Up Mock and the Environment

The first phase in the process is to set up Mock and the environment for building the RPMs. Fortunately, this is relatively simple.

First, activate EPEL. My preferred method for activating the EPEL repository is to download the RPM, then use yum localinstall to install it, like this:

wget http://fedora.mirrors.pair.com/epel/6/i386/\
epel-release-6-8.noarch.rpm
yum localinstall epel-release-6-8.noarch.rpm

(Note that I’ve line-wrapped the URL with a backslash to make it more readable. That line-wrapped command actually works just as it is in the shell as well.)

Next, you’ll need to install Mock and related RPM-building tools:

yum install fedora-packager

Third, create a dedicated user for building RPMs. I use “makerpm” as my username, but you could use something else. Just make sure that the name makes sense to you, and that the new user is a member of the mock group:

useradd makerpm -G mock
passwd makerpm

From this point on, you’ll want to be running as this user you just created, so switch to that user with su - makerpm. This ensures that the RPMs are built under this dedicated user account.

The final step in setting up Mock and the build environment is to run the following command while running as the dedicated user you created:

rpmdev-setuptree

Now you’re ready to move on to the next phase: installing prerequisites into the Mock environment.

Installing Prerequisites Into the Mock Environment

One of the great things about Mock is that it creates an isolated chroot into which it installs all the necessary prerequisites for a particular package. This helps ensure that the package’s dependencies are managed correctly. However, if you are trying to build a package where dependencies don’t exist in the repositories, then you have to take a few additional steps. When you’re trying to build libvirt 1.0.1 RPMs for use with CentOS 6.3, you’ll find yourself in exactly this situation. Libvirt has dependencies on newer versions of sanlock-devel and libssh2-devel than are available in the repositories.

Fortunately, there is a workaround—and here’s where those other posts on Mock will come in handy. Use the instructions posted here to build RPMs for sanlock 2.4, and use the instructions here to build RPMs for libssh2 1.4.1.

Once the RPMs are built (and they should build without any major issues, based on my testing), then use these commands to get them into the isolated Mock environment (I’ve line-wrapped here with backslashes for readability):

mock -r epel-6-x86_64 --init
mock -r epel-6-x86_64 --install \
~/rpmbuild/RPMS/sanlock-lib-2.4-3.el6.x86_64.rpm \
~/rpmbuild/RPM/sanlock-devel-2.4-3.el6.x86_64.rpm \
~/rpmbuild/RPMS/libssh2-1.4.1-2.el6.x86_64.rpm \
~/rpmbuild/RPMS/libssh2-devel-1.4.1-2.el6.x86_64.rpm

This will install these packages into the Mock environment, not onto the general Linux installation.

Once you’ve gotten these packages compiled and installed, then you’re ready for the final phase: building the libvirt RPMs.

Building the Libvirt RPMs

As in my earlier post on compiling Libvirt 1.0.1 RPMs for CentOS 6.3, this final step is almost anti-climactic. That’s good, though, because it means you’ve done all the previous steps perfectly.

First, fetch the source RPM from the libvirt HTTP server:

wget http://libvirt.org/sources/libvirt-1.0.1-1.fc17.src.rpm

Next, move the source RPM into the ~/rpmbuild/SRPMS directory:

mv libvirt-1.0.1-1.fc17.src.rpm ~/rpmbuild/SRPMS

Finally, run Mock to rebuild the RPMs:

mock -r epel-6-x86_64 --no_clean ~/rpmbuild/SRPMS/libvirt-1.0.1-1.fc17.src.rpm

Note that the --no-clean parameter is required here to prevent Mock from cleaning out the chroot and getting rid of the packages you installed into the environment earlier.

This command should run without any errors or problems, and produce a set of RPMs (typically) found in /var/lib/mock/epel-6-x86_64/results. You can then take these RPMs and install them on another CentOS 6.3 system using yum localinstall.

Testing the RPMs

To verify that everything worked as expected, I tested the RPMs using these steps:

  1. Using a clean CentOS 6.3 VM (built using the “Minimal Desktop” option), I used yum groupinstall to install the Virtualization, Virtualization Client, Virtualization Platform, and Virtualization Tools groups. This installed version 0.9.10 of libvirt.

  2. I then installed the updated version of libvirt using yum localinstall. I had to specify the dependencies manually on the command line; I anticipate that had I been using a real repository, this would not have been the case. The updated libvirt, sanlock, and libssh2 packages all installed correctly.

  3. I started the libvirtd service (it worked), and ran virsh --version. It returned version 1.0.1.

I imagine there might be more comprehensive/better ways of testing the RPMs that I built, but they seemed to work fine on my end. If anyone has any other suggestions for how we can double-check to ensure the packages are working correctly, feel free to speak up in the comments below. I also welcome any other corrections, suggestions, or questions in the comments. Courteous comments are always welcome.

Tags: , , , , ,

In previous articles, I’ve shown you how to compile libvirt 0.10.1 on CentOS 6.3, but—as several readers have pointed out in the comments to that and other articles—compiling packages from source may not be the best long-term approach. Not only does it make it difficult to keep the system up-to-date, it also makes automating the configuration of the host rather difficult. In this post, I’ll show you how to rebuild a source RPM for libvirt 1.0.1 so that it will install (and work) under CentOS 6.3. (These instructions should work for RHEL 6.3, too, but I haven’t tested them.)

Overview

The process for rebuilding a source RPM isn’t too terribly difficult, assuming that you can get the dependencies worked out. Here’s a quick look at the steps involved:

  1. Create a set of build directories for source RPMs.
  2. Download the source RPM and install all prerequisites/dependencies onto the system.
  3. Rebuild the source RPM for the destination system.

Let’s take a look at each of these steps in a bit more detail.

Create the Build Environment

The CentOS wiki has a great page for how to set up an RPM build environment. I won’t repeat all the steps here (refer to the wiki page instead), but here’s a quick summary of what’s involved:

  1. Install the necessary packages (typically you’ll need the rpmbuild, redhat-rpm-config, make, and gcc packages). You might also need certain development libraries, but this will vary according the source RPMs you’re rebuilding (more on that in the next section).
  2. Create the necessary directories under your home directory.

I’ll assume that you’ve followed the steps outlined in the CentOS wiki to set up your environment appropriately before continuing with the rest of this process.

Download the Source RPM and Install Prerequisites

The libvirt 1.0.1 source RPMs are available directly from the libvirt HTTP server, easily downloaded with wget:

wget http://libvirt.org/sources/libvirt-1.0.1-1.fc17.src.rpm

You can just download the source RPM to your home directory. Before you can build a new RPM from the source RPM, though, you’ll first need to install all the various prerequisites that libvirt requires. Most of them can be installed easily using yum with a command like this:

yum install xhtml1-dtds augeas libudev-devel \
libpci-access-devel yajl-devel, libpcap-devel libnl-devel \
avahi-devel radvd ebtables qemu-img iscsi-initiator-utils \
parted-devel device-mapper-devel numactl-devel netcfg-devel \
systemtap-sdt-devel scrub numad libblkid-devel

There are two dependencies, though—sanlock and libssh2—that require versions newer than what are available in the CentOS/EPEL repositories. For those, you’ll need to recompile your own RPMs. Fortunately, this is pretty straightforward. The next two sections provide more details on getting these prerequisites handled.

Building an RPM for sanlock

To build a CentOS 6.3 RPM for version 2.4 of sanlock (the minimum version needed by libvirt 1.0.1), first use wget to download a Fedora 17 version of the source RPM. I’ve wrapped the URL with a backslash for readability:

wget http://dl.fedoraproject.org/pub/fedora/linux/updates/17/SRPMS\
/sanlock-2.4-3.fc17.src.rpm

Next, install an prerequisite library using yum install libaio-devel.

Finally, use rpmbuild to rebuild the sanlock source RPM:

rpmbuild --rebuild sanlock-2.4-3.fc17.src.rpm

This process should proceed without any problems. The resulting RPMs that are created will be found in ~/rpmbuild/RPMS/x86_64 (assuming you are, as I am, using a 64-bit build of CentOS).

Building the RPMs, however, isn’t enough—you need to install them so that you can build the libvirt RPMs. So install the sanlock-devel and sanlock-lib RPMs using yum locainstall (do this command from the directory where the RPMs reside):

yum localinstall sanlock-devel-* sanlock-lib-*

That should take care of the sanlock dependency.

Building an RPM for libssh2

To build a CentOS 6.3 RPM for version 1.4.1 of libssh2 (libvirt 1.0.1 requires at least version 1.3.0), first download the source RPM using wget (I’ve wrapped the URL here for readability):

wget http://dl.fedoraproject.org/pub/fedora/linux/releases\
/17/Everything/source/SRPMS/l/libssh2-1.4.1-2.fc17.src.rpm

(That’s a lowercase L in the URL just after SRPMS.)

Once you have the source RPM downloaded, then just rebuild the source RPM:

rpmbuild --rebuild libssh2-1.4.1-2.fc17.src.rpm

Then install the resulting RPMs using yum localinstall:

yum localinstall libssh2-1.4.1-* libssh2-devel-*

That takes care of the last remaining dependency. You’re now ready to compile the RPMs for libvirt 1.0.1.

Build the Libvirt RPM

This part is almost anticlimactic. Just use the rpmbuild command:

rpmbuild --rebuild libvirt-1.0.1-1.fc17.src.rpm

If you’ve successfully installed all the necessary prerequisites, then the RPM compilation process should proceed without any issues.

Once the RPM compilation process is complete, you’ll find libvirt 1.0.1 RPMs in the ~/rpmbuild/RPMS/x86_64 directory (assuming a 64-bit version of CentOS) which you can easily install with yum localinstall, or post to your own custom repository.

I hope this post helps someone. If you have any questions, or if you spot an error, please speak up in the comments below. All courteous comments are welcome!

Tags: , , , ,

Welcome to Technology Short Take #27! This is my usual collection of links, thoughts, rants, and ideas about data center-related technologies. Here’s hoping you find something useful!

Networking

  • If you’re interested in learning more about OpenFlow and software-defined networking but need to do this on a shoestring budget in your home lab, a number of guides have been written to help out. I haven’t personally used any of these guides yet, but I’m working my way in that direction. (I needed to fill in some other knowledge gaps first.) First up is Brent Salisbury’s how to build an SDN lab without needing OpenFlow hardware. Brent is creating some fantastic content that I’ve found extremely useful. His earlier post on getting started with OpenFlow and Open vSwitch tutorial lab is also quite good. Another good resource is Dan Hersey’s guide to building an SDN-based private cloud in an hour. I encourage you to have a look at these posts if you’re at all interested in any of these technologies.
  • Bruce Davie and Martin Casado (with Nicira, now part of VMware) have written a post comparing the VXLAN and STT tunneling protocols. Not unsurprisingly, one of the key advantages of STT that’s highlighted is its improved performance due to TSO support in NIC hardware. VXLAN, on the other hand, is seeing broader adoption across multiple vendors. There’s no mention of NVGRE (or just plain GRE).
  • Related to the bare metal provisioning work (see below under “Servers/Hardware”), Mirantis also detailed some bare-metal networking stuff they’ve done for OpenStack in relation to the use of bare metal nodes.

Servers/Hardware

  • Mirantis published an article discussing a framework they built for bare-metal provisioning with OpenStack that allows OpenStack to place workloads onto bare-metal nodes instead of onto a hypervisor. It’s interesting work, but unfortunately it looks like this work won’t be returned to the community (it was developed for one or more of their clients). There are also a few follow-up posts, such as this one on placement control and multi-tenancy isolation and this one on preparing images for bare metal nodes. Also see the “Networking” section above for a related post on the networking aspects involved.

Security

I don’t have anything for this area this time around, but I’ll stay alert for articles to add next time. Feel free to share something in the comments!

Cloud Computing/Cloud Management

  • I might have mentioned this before, but Ken Pepple’s OpenStack Folsom architecture post is just awesome. It’s well worth reading and reviewing in depth.
  • This OpenStack-on-Debian HOWTO is a bit older (and probably out of date), but it does give a decent overview of the components that are involved and—via the configuration—how they relate to each other. While the details for installing a current version of OpenStack are likely to be different now, you might still find this conceptually helpful.
  • These articles are a bit long in the tooth, but CSS Corp has a useful series of articles on bundling various Linux distributions for use with OpenStack: bundling CentOS, bundling CentOS with VNC, bundling Debian, and bundling OpenSUSE. It would be interesting to me to see how much of this, if any, could be automated with something like Puppet. If any enterprise Puppet experts want to give it a go, I’d be happy to publish a guest blog post for you with full details on how it’s done.
  • Much like there are some great “how to’s” on how to run an SDN lab (see the Networking section earlier), there are also some great write-ups on doing the same for OpenStack. For example, Cody Bunch published this article on running OpenStack Private Cloud on ESXi, and Brent Salisbury (there he is again!) posted an older guide to OpenStack Essex on Ubuntu on VirtualBox as well as a newer guide to OpenStack DevStack on Fusion.

Operating Systems/Applications

Storage

  • I don’t fully understand all the details involved, but this post on changes in block protocol scalability in Xen outlines what sounds like good progress in improving efficiency.
  • This article is a bit older, published at the start of October, but it talks about an interesting project (product?) by Qlogic called “Mt. Rainier.” (Stu Miniman of Wikibon has more information here as well.) Apparently, “Mt. Rainier” will allow customers to combine PCIe-based SSD storage inside servers into a “virtual SAN” (now there’s an original and not over-used term). The really interesting aspect, in my opinion, is the use of “Mt. Rainier” to create shared caches across servers. Is this the beginning of the data center fractal edge?

Virtualization

  • Big news in the QEMU world: In the QEMU 1.3 release, the QEMU-KVM and QEMU projects have been merged. Why is this important? It’s first necessary to understand the relationship between QEMU and KVM. KVM is the set of kernel modules that leverage hardware virtualization functionality inside Intel and AMD CPUs, and it makes possible the virtualization of closed-source operating systems like Windows. QEMU, on the other hand, is needed to emulate everything else that a VM needs: networking, storage, USB, keyboard, mouse, etc. Both KVM and QEMU are needed for a full virtualization solution. Until the 1.3 release, QEMU (without hardware acceleration via KVM) was one branch, and QEMU-KVM (with KVM hardware acceleration) was a separate branch. The QEMU 1.3 release completes an effort to merge both efforts into a single development tree.
  • The merge of QEMU and QEMU-KVM isn’t the only cool thing happening with QEMU; also included in the 1.3 release is GlusterFS integration. This integration dramatically improves GlusterFS performance by allowing QEMU’s block layer to communicate directly with the Gluster backend without going through the userspace FUSE components.
  • Erik Scholten of VMGuru.nl has posted a good hypervisor feature comparison document. It includes RHEV 3.1 in the comparison, even though RHEV 3.1 wasn’t released (was still in beta) at the time the comparison was written.
  • Speaking of RHEV: apparently RHEV 3.1 was released yesterday (Wednesday, December 4, 2012), although I haven’t been able to find any sort of official press release or announcement.
  • Debunking an argument I’ve heard quite a bit is this article by Frank Denneman on using SIOC with multiple datastores backed by a single pool of disks.
  • Need to compact a virtual hard disk in Windows 8/Windows Server 2012? Ben Armstrong shows how here.
  • I enjoyed this article by Josh Townsend on using SUSE Studio and HAProxy to create a (free) open source load balancing solution for VMware View.

That’s it for this time around; no need to overwhelm you with too much information! Besides, I have to keep a few items around for Technology Short Take #28…

As always, comments, thoughts, rants, or corrections are welcome below.

Tags: , , , , , , , , , , , , ,

In my post on installing KVM and Open vSwitch on Ubuntu, I mentioned that I was using libvirt as the virtualization API/toolkit by which to manage KVM. Unfortunately, that particular version of libvirt didn’t have built-in support for Open vSwitch; that was added in libvirt 0.9.11 (I think that I was using 0.9.8 in my testing). In any case, I noticed that libvirt had released 0.10.1 on August 31, and it included a couple of OVS fixes regarding VLANs and VLAN support (among a host of other fixes; see here for more details). Considering that OVS is a key focus area for me (it’s foundational to OpenStack Quantum), I thought I’d upgrade my test system to the latest version of libvirt. Because the release was so new, there weren’t yet any precompiled binary packages of the latest version, which meant I had to compile and install it on my own.

My first attempt to do so was using Ubuntu 12.04.1 LTS, and while it appeared successful at first, I soon found that it was horribly broken. While I do really like Ubuntu as a desktop Linux distribution, I’m not entirely convinced (and even less so now) about it’s place as a server distribution. With that in mind, and considering the enormous popularity of Red Hat, I decided to rebuilt my test system with CentOS 6.3 (the “community version” of Red Hat Enterprise Linux 6.3) and try it again there. In order to use libvirt, I naturally had to get KVM installed on CentOS 6.3 first; here is the process I used.

Once KVM was installed, this attempt at compiling the latest version of libvirt was much more successful, so I wanted to document the process for others.

I started out with a CentOS 6.3 x86_64 system that already had KVM and libvirt installed from packages (see this article for full details), and from there I followed these steps:

  1. First I downloaded the tarball for libvirt 0.10.1 from the libvirt.org HTTP server. You could probably use curl or wget to do this directly from the CentOS server; in my case, I downloaded it on my Mac and then used SFTP to push it to the CentOS server.

  2. Next, I extracted the files using this command:

  3. gunzip -c libvirt-0.10.1.tar.gz | tar xvf -
  4. I used cd to change into the newly-created libvirt-0.10.1 directory that resulted from the commands in step 2.

  5. Through process of elimination (in other words, running configure again and again) I determined that the following packages need to be installed before you’ll be able to successfully compile libvirt 0.10.1: gcc, make, libxml2-devel, gnutls-devel, device-mapper-devel, python-devel, and libnl-devel. I used yum install to install each of these packages.

  6. With all the dependencies now satisfied, I ran the configure command from the extracted libvirt-0.10.1 directory. I selected the specified directories because that matched where the existing binaries were on the system (there’s probably an easier way):

  7. ./configure --prefix=/usr --localstatedir=/var --sysconfdir=/etc
  8. The above command results in a libvirt that supports KVM, OpenVZ, LXC, etc., but doesn’t support Xen or ESX. You’d have to add the --with-xen or --with-esx parameters to the configure command to include support for those. Following completion of the previous command, it was very straightforward:

  9. make
    make install
    ldconfig
  10. Finally, I restarted libvirtd with service libvirtd restart. The daemon restarted on my system cleanly and without any errors.

Failing to run ldconfig at the end, by the way, resulted in numerous errors when trying to run virsh.

From there, I was able to run libvirtd --version or virsh --version to verify that the system was, in fact, running libvirt 0.10.1. All in all, it was pretty straightforward, and I was pleasantly surprised that it wasn’t more complicated or troublesome.

Now, if only the story with Open vSwitch and CentOS 6.3 was as simple…but that is a post for another day. Until then, feel free to speak up in the comments below.

Tags: , , , , ,

After spending some time working with KVM on Ubuntu (see this post), I thought it might be worthwhile to try the same thing on a different Linux distribution. I like Ubuntu (generally), but wanted to try it on a Red Hat/CentOS system. So, here’s a look at installing KVM on CentOS 6.3. To be honest, it’s actually pretty simple. For another look at the process, see this HowtoForge post.

For reasons that will become clear in a future post, I did not install Open vSwitch during this process. (Short story: There’s a known bug in Open vSwitch caused by a backport of a kernel fix to the kernel version used by RedHat/CentOS 6.3, and I haven’t been able to find a fix yet.)

I started this process with a minimal install (the default option) of 64-bit CentOS 6.3.

First, use yum groupinstall (handy feature, by the way) to install the virtualization-related packages (line-wrapped here for readability):

yum groupinstall Virtualization "Virtualization Client" \
"Virtualization Platform" "Virtualization Tools"

This page breaks down these four groups and lists the individual packages contained in each.

However, the libvirtd daemon wouldn’t start after this process. In reviewing the log files (found, by default, at /var/log/libvirt), I found that it was failing due to a problem with multicast DNS (mDNS). That was fixed with:

yum install avahi
service start avahi-daemon

This site alluded to the need for avahi to be installed, but I was a bit surprised that it didn’t get installed automatically during the yum groupinstall process. Once avahi was installed, libvirtd started cleanly. I was then able to run virsh without any issues or errors.

Normally, from here you’d continue with setting up a Linux bridge, etc. I stopped here with the intention of first upgrading libvirt to the latest build and then installing Open vSwitch, but there are plenty of other “how to’s” that outline any additional follow-up steps.

I hope this helps someone!

Tags: , , , ,

Since I had CentOS 5 up and running on ESX Server in the test lab, I decided to try to validate my latest Linux-AD integration instructions on this installation.  Unfortunately, the instructions do not seem to work well at all with CentOS 5; here are some of the errors that I ran into:

  • When using “net ads join” to join the Active Directory domain, it didn’t recognize any existing Kerberos tickets.  I’d already run a “kinit <AD username>”, but “net ads” continued to either a) try to use the root account if I didn’t specify the “-U <AD username>” parameter, and b) prompt for password even when I’d already obtained a Kerberos ticket for the specified username.
  • When initially trying to join the Active Directory domain, “net ads join” threw this error:
    [2007/12/04 12:57:08, 0] libads/kerberos.c:create_local_private_krb5_conf_
    for_domain(594) create_local_private_krb5_conf_for_domain:
    failed to create directory /var/cache/samba/smb_krb5.
    Error was Permission denied

    This error persisted until I manually created the /var/cache/samba/smb_krb5 directory myself.  Why this directory wasn’t created automatically during the Samba installation is beyond me.  Once I created that directory, the error went away, but Samba still wouldn’t create the keytab or add entries to the keytab.
  • The “net ads keytab” command failed miserably; it would not create a keytab, nor would it add entries to a keytab.  No error message is reported; it just doesn’t work.

I inquired on the #samba IRC channel on irc.freenode.net, but the only person willing or able to respond didn’t have any information to provide (in fact, he’d actually used my Solaris-AD integration instructions as a guide for some of his own work).  Various Google searches also failed to provide any helpful information.

By the way, these tests were performed on a stock installation of CentOS 5, with all the latest packages installed using “yum update”.  The Samba version was 3.0.25b-1.el5_1.2.

In the end, I’ve given up on trying to make Samba work in the AD integration process and will instead fallback to the use of ktpass.exe to create the keytab file.  If you have any useful information to share, please let me know or post it in the comments.  Thanks!

Tags: , , ,

CentOS 5 on ESX Server

As I had to rebuild some Linux VMs in the lab anyway (they’d gotten messed up with various interoperability tests), I decided to try version 5.0 of CentOS.  I was trying to get this done on Friday before the long holiday weekend, but it turns out that downloading the ISO images for six (yes, six!) CD-ROMs takes a bit longer than I had hoped.  So I downloaded them at home on my 6Mbit DSL connection over the weekend, and had the opportunity to work on it yesterday while at the office.

I didn’t really expect any major issues that would prevent the new version of the distribution from running, but you never really know for sure until you try.  Fortunately, CentOS 5.0 loaded quickly and without any real problems on the lab servers running ESX Server version 3.0.2.  The only issue that came up—and it was a minor one, really, more caused by my own lack of preparation than anything else—was that the VMware Tools did not have a suitable kernel module or a suitable VMXNet driver.  This was only a problem at first because I didn’t have the right parts (compiler, kernel header files, etc.) installed to allow the VMware Tools installer to create its own module and driver.  I had to use yum to install the necessary pieces, then the VMware Tools installer worked without any further problems.

Later in the week I hope to be able to post some information on running Solaris 10 8/07 (aka Update 4) on ESX Server as well.

Tags: , , , ,

In the event that your organization is considering a migration later this year (or next?) to Windows Server 2008 (formerly “Longhorn”), here are some instructions for integrating Linux login requests against Active Directory on Windows Server 2008. These instructions are based on Linux-AD Integration, Version 4 and utilize Kerberos, LDAP, and Samba.

When this process is complete, AD users can be enabled for use on Linux systems on the network and login to those Linux systems using the same username and password as throughout the rest of Active Directory.

If you are looking for information on using Linux with a previous version of Windows before Windows Server 2008, please refer back to my AD integration index and select the appropriate article. The only significant changes in the process involve the mapping of the LDAP attributes; otherwise, the procedure is very similar between the two versions of Windows.

Preparing Active Directory (One-Time)

The process of installing and configuring Windows Server 2008 is beyond the scope of this article (although I may touch on that in the near future in a separate article). Therefore, I won’t provide detailed instructions on how to perform some of these tasks, but instead provide a high-level overview.

Enable Editing/Display of UNIX Attributes

In order to store UNIX attributes in Active Directory, the schema must be extended. To extend the schema, first install Active Directory (add the Active Directory Domain Services role to an installed server, then use the Active Directory Installation Wizard to setup Active Directory) and then add the “Identity Management for UNIX” role service (this can be done in Server Manager).

Once that role service has been installed, then the AD schema now includes a partially RFC 2307-compliant set of UNIX attributes, such as UID, UID number, GID number, login shell, etc. (Note that it may be that these attributes are already included in the schema for Windows Server 2008; I did not check the schema before installing the Identity Management for UNIX role service. With Windows Server 2003 R2, the schema was present at the time of installation, but the attributes were not visible until installing the UNIX identity services.)

At this point a new tab labeled “UNIX Attributes” will appear in the properties dialog box for users and groups in Active Directory. You’ll use this tab to edit the UNIX-specific attributes that are required for logins to Linux-based systems.

Create an LDAP Bind Account

You’ll also need to create an account in Active Directory that will be used to bind to Active Directory for LDAP queries. This account does not need any special privileges; in fact, making the account a member of Domain Guests and not a member of Domain Users is perfectly fine. This helps minimize any potential security risks as a result of this account. Just be sure that you know the account’s user principal name (UPN) and password.

Prepare Active Directory (Each User)

Each Active Directory account that will authenticate via Linux must be configured with a UID and other UNIX attributes. This is accomplished via the new “UNIX Attributes” tab on the properties dialog box of a user account.

After all the user accounts have been configured, then we are ready to configure Active Directory objects for each of the Linux server(s) that we’ll be integrating with AD.

Prepare Active Directory (Each Server)

Prior to using Samba to join Linux computers to Active Directory and generate a keytab automatically, we had to use the ktpass.exe utility on Windows to generate a keytab. Due to some current Samba-Windows Server 2008 interoperability issues, we can’t use Samba. That means we’ll be back to using ktpass.exe to map service principals onto accounts in Active Directory. Unfortunately, you’ll need to first disable User Account Control (UAC) on your server, since UAC interferes with ktpass.exe. (Nice, huh?)

Once you’ve disabled UAC (and rebooted your server), then you can map the service principal names (SPNs) using the following steps:

  1. Create a computer account (or a user account; either will work) with the name of the Linux server.
  2. Use the following command to map the needed SPN onto this account (backslashes indicate line continuation):
    ktpass.exe -princ HOST/[email protected] \
    -mapuser DOMAIN\AccountName$ -crypto all \
    -pass Password123 -ptype KRB5_NT_PRINCIPAL \
    -out filename.keytab
  3. Copy this file to the Linux server (using SCP or SFTP is a good option) and merge it with the existing keytab (if it exists) using ktutil. If there is no existing keytab, simply copy the file to /etc/krb5.keytab and you should be good to go.

Now that Active Directory has computer objects (and, more importantly, SPNs) for the Linux servers and the AD users have been enabled for UNIX (by populating the UNIX attributes), we’re ready to start configuring the Linux server(s) directly.

Prepare Each Linux Server

Follow the steps below to configure the Linux server for authentication against Active Directory. (Note that this configuration was tested on a system running CentOS—a variation of Red Hat Enterprise Linux—version 4.3.)

  1. Edit the /etc/hosts file and ensure that the server’s fully-qualified domain name is listed first after its IP address.
  2. Make sure that the appropriate Kerberos libraries, OpenLDAP, pam_krb5, and nss_ldap are installed. If they are not installed, install them.
  3. Be sure that time is being properly synchronized between Active Directory and the Linux server in question. Kerberos requires time synchronization. Configure the NTP daemon if necessary.
  4. Edit the /etc/krb5.conf file to look something like this, substituting your actual host names and domain names where appropriate:
    [logging]
    default = FILE:/var/log/krb5libs.log
    kdc = FILE:/var/log/krb5kdc.log
    admin_server = FILE:/var/log/kadmind.log
     
    [libdefaults]
    default_realm = EXAMPLE.COM
    dns_lookup_realm = true
    dns_lookup_kdc = true
     
    [realms]
    EXAMPLE.COM = {
    kdc = host.example.com:88
    admin_server = host.example.com:749
    default_domain = example.com
    }
     
    [domain_realm]
    .example.com = EXAMPLE.COM
    example.com = EXAMPLE.COM
     
    [kdc]
    profile = /var/kerberos/krb5kdc/kdc.conf
     
    [appdefaults]
    pam = {
    debug = false
    ticket_lifetime = 36000
    renew_lifetime = 36000
    forwardable = true
    krb4_convert = false
    }
  5. Edit the /etc/ldap.conf file to look something like this, substituting the appropriate host names, domain names, account names, and distinguished names (DNs) where appropriate. (Please note that the nss_base_group line should not be broken across two lines when you edit it; it has been wrapped here for readability with the line continuation noted by a backslash)
    host 10.10.10.10
    base dc=example,dc=com
    uri ldap://server.example.com/
    binddn [email protected]
    bindpw adldapbindpw
    scope sub
    ssl no
    nss_base_passwd dc=example,dc=com?sub
    nss_base_shadow dc=example,dc=com?sub
    nss_base_group dc=mydomain,dc=com?sub? \
    &(objectCategory=group)(gidnumber=*)
    nss_map_objectclass posixAccount user
    nss_map_objectclass shadowAccount user
    nss_map_objectclass posixGroup group
    nss_map_attribute gecos cn
    nss_map_attribute homeDirectory unixHomeDirectory
    nss_map_attribute uniqueMember member
  6. Configure PAM (this varies according to Linux distributions) to use pam_krb5 for authentication. Many modern distributions use a stacking mechanism whereby one file can be modified and those changes will applied to all the various PAM-aware services. For example, in Red Hat-based distributions, the system-auth file is referenced by most other PAM-aware services. Here’s a properly edited /etc/pam.d/system-auth file taken from CentOS 4.4 (some lines have been wrapped for readability, as noted by a backslash; do not wrap them when editing the file):
    #%PAM-1.0
    # This file is auto-generated.
    # User changes will be destroyed the next time
    # authconfig is run.
    auth required /lib/security/pam_env.so
    auth sufficient /lib/security/pam_unix.so \
       likeauth nullok
    auth sufficient /lib/security/pam_krb5.so
    auth required /lib/security/pam_deny.so
     
    account sufficient /lib/security/pam_unix.so
    account sufficient /lib/security/pam_krb5.so
    account sufficient /lib/security/pam_succeed_if.so \
       uid < 100 quiet
    account required /lib/security/pam_deny.so
     
    password requisite /lib/security/pam_cracklib.so \
       retry=3
    password sufficient /lib/security/pam_unix.so
       nullok use_authtok md5 shadow \
    password required /lib/security/pam_deny.so
     
    session required /lib/security/pam_limits.so
    session required /lib/security/pam_unix.so
  7. Edit the /etc/nsswitch.conf file to include “ldap” as a lookup source for passwd, shadow, and groups.

At this point we are now ready to test our configuration and, if successful, to perform the final step: to join the Linux server to Active Directory for authentication.

Test the Configuration

To test the Kerberos authentication, use the kinit command, as in kinit <AD username>@<AD domain DNS name>. This should return no errors. A klist at that point should then show that you have retrieved a TGT (ticket granting ticket) from the AD domain controller. If this fails, go back and troubleshoot the Kerberos configuration. In particular, if you are seeing references to failed TGT validation, check to make sure that both your Linux servers and AD domain controllers have reverse lookup (PTR) records in DNS and that the Linux server’s /etc/hosts file listed the FQDN of the server first instead of just the nodename.

<aside>Some readers and some other articles have suggested the use of the AD domain DNS name in the /etc/krb5.conf file instead of an AD domain controller specifically; I recommend against this. First, I believe it may contribute to TGT validation errors; second, it is possible to list multiple KDCs (AD DCs) in the configuration. Since the only major reason to use the AD domain DNS name instead of the DNS name of one or more DCs would be fault tolerance, then it doesn’t really gain anything.</aside>

To test the LDAP lookups, use the getent command, as in getent passwd <AD username>. This should return a listing of the account information from Active Directory. If this does not work, users will not be able to login, even if Kerberos is working fine. If you run into errors or failures here, go back and double-check the LDAP configuration. One common source of errors is the name of the LDAP bind account, so be sure that is correct.

At this point, SSH logins to the Linux system using an account present in Active Directory (one which has had its UNIX attributes specified properly) should be successful. This will be true as long as you used the ktpass.exe command earlier to map the SPN onto the computer object in Active Directory. Even if you didn’t copy the keytab over to the Linux server, logins will work. Why? Because the PAM Kerberos configuration, by default, does not require a client keytab, and does not attempt to validate the tickets granted by the TGT. This means that as long as the SPN(s) are mapped to the accounts in AD, the keytab is not necessarily required.

(Note, however, that not using a keytab and/or not requiring a keytab does leave the Linux server open to potentially spoofed Kerberos tickets from a fake KDC. In addition, “native” Kerberos authentication—i.e., using a Kerberos ticket to authenticate instead of typing in a password—won’t work without a keytab.)

Deal with Home Directories

Unlike Windows systems, home directories are required on Linux-based systems. As a result, we must provide home directories for each AD user that will log in to a Linux-based system. We basically have three options here:

  • Manually create home directories and set ownership/permissions properly before users will be able to log in.
  • Use the pam_mkhomedir.so PAM module to automatically create local home directories “on the fly” as users log in. To do this, you would add an entry for pam_mkhomedir.so in the session portion of the PAM configuration file.
  • Use the automounter to automatically mount home directories from a network server. This process is fairly complicated (too involved to include the information here), so I’ll refer you to this article on using NFS and automounts for home directories. This has the added benefit of providing a foundation for unified home directories across both Windows and Linux systems.

Once you’ve settled on and implemented a system for dealing with home directories, you are finished! UNIX-enabled users in Active Directory can now login to Linux-based systems with their Active Directory username and password.

What’s not addressed in this article? Password management. In this configuration, users will most likely not be able to change their password from the Linux servers and have that change properly reflected in Active Directory. In addition, “native” Kerberos authentication using Kerberos tickets won’t work unless the keytab is present. In my testing, I ran into a number of issues with the keytab and TGT validation, but I’m not sure if those are errors in my process or the result of the beta status of Windows Server 2008.

I welcome your corrections, additions, or suggestions in the comments below.

Tags: , , , , , , ,

This procedure allows Linux-based systems to authenticate against Active Directory. We use Kerberos for authentication, LDAP for account information, and Samba to help automate the process along the way. When this process is complete, AD users can be enabled for use on Linux systems on the network and login to those Linux systems using the same username and password as throughout the rest of Active Directory.

These instructions are designed for use with Windows Server 2003 R2. If you are looking for information on using Linux with a previous version of Windows, please refer back to this article. The only significant changes in the process involve the mapping of the LDAP attributes; otherwise, the procedure is very similar between the two versions of Windows.

Preparing Active Directory (One-Time)

Enable Editing/Display of UNIX Attributes

Based on my research, it appears that the partially RFC 2307-compliant schema is installed with Windows Server 2003 R2; this means that the schema does not need to be extended to include UNIX-specific attributes such as uid, gid, login shell, etc. However, while the attributes are there in the schema, there is no way to edit those attributes, and these attributes must be populated correctly in order for this process to work.

The easiest way to enable the editing of these attributes is to install the “Server for NIS” component on at least one domain controller. This will cause a new tab, labeled “UNIX Attributes,” to appear in the properties dialog box for users and groups. You’ll use this tab to edit the UNIX-specific attributes that are required for logins to Linux-based systems. Please note that due to the way this tab is displayed, you’ll need Schema Admin privileges in order to install the “Server for NIS” component on your domain controller. (More information on this issue is available here.)

You could just as well use LDP, LDIF files, ADSI Edit, or any number of other methods to display and edit these attributes. To make this process as seamless as possible, however, you’ll want to integrate the management of these attributes into Active Directory Users and Computers using the method described above.

Create an LDAP Bind Account

You’ll also need to create an account in Active Directory that will be used to bind to Active Directory for LDAP queries. This account does not need any special privileges; in fact, making the account a member of Domain Guests and not a member of Domain Users is perfectly fine. This helps minimize any potential security risks as a result of this account.

Prepare Active Directory (Each User)

Each Active Directory account that will authenticate via Linux must be configured with a UID and other UNIX attributes. This is accomplished via the new “UNIX Attributes” tab on the properties dialog box of a user account. (Installing the “Server for NIS” component enables this, as mentioned previously.)

After all the user accounts have been configured, then we are ready to configure the Linux server(s) for authentication against Active Directory.

Prepare Each Linux Server

Follow the steps below to configure the Linux server for authentication against Active Directory.

  1. Edit the /etc/hosts file and ensure that the server’s fully-qualified domain name is listed first after its IP address.
  2. Make sure that the appropriate Kerberos libraries, OpenLDAP, pam_krb5, and nss_ldap are installed. If they are not installed, install them.
  3. Be sure that time is being properly synchronized between Active Directory and the Linux server in question. Kerberos requires time synchronization. Configure the NTP daemon if necessary.
  4. Edit the /etc/krb5.conf file to look something like this, substituting your actual host names and domain names where appropriate:
    [logging]
     default = FILE:/var/log/krb5libs.log
     kdc = FILE:/var/log/krb5kdc.log
     admin_server = FILE:/var/log/kadmind.log
    
    [libdefaults]
     default_realm = EXAMPLE.COM
     dns_lookup_realm = true
     dns_lookup_kdc = true
    
    [realms]
     EXAMPLE.COM = {
      kdc = host.example.com:88
      admin_server = host.example.com:749
      default_domain = example.com
     }
    
    [domain_realm]
     .example.com = EXAMPLE.COM
     example.com = EXAMPLE.COM
    
    [kdc]
     profile = /var/kerberos/krb5kdc/kdc.conf
    
    [appdefaults]
     pam = {
       debug = false
       ticket_lifetime = 36000
       renew_lifetime = 36000
       forwardable = true
       krb4_convert = false
     }
  5. Edit the /etc/ldap.conf file to look something like this, substituting the appropriate host names, domain names, account names, and distinguished names (DNs) where appropriate. (Please note that the nss_base_group line should not be broken across two lines when you edit it; it has been wrapped here for readability.) (Note: These schema mappings assume that you are using the newer schema extensions provided by Windows Server 2003 R2. If you are using SFU 3.5 instead, you will need to use the schema mappings described here.)
    host 10.10.10.10
    base dc=example,dc=com
    uri ldap://server.example.com/
    binddn [email protected]
    bindpw adldapbindpw
    scope sub
    ssl no
    nss_base_passwd dc=example,dc=com?sub
    nss_base_shadow dc=example,dc=com?sub
    nss_base_group dc=mydomain,dc=com?sub?
        &(objectCategory=group)(gidnumber=*)
    nss_map_objectclass posixAccount user
    nss_map_objectclass shadowAccount user
    nss_map_objectclass posixGroup group
    nss_map_attribute gecos cn
    nss_map_attribute homeDirectory unixHomeDirectory
    nss_map_attribute uniqueMember member
  6. Configure PAM (this varies according to Linux distributions) to use pam_krb5 for authentication. Many modern distributions use a stacking mechanism whereby one file can be modified and those changes will applied to all the various PAM-aware services. For example, in Red Hat-based distributions, the system-auth file is referenced by most other PAM-aware services. Here’s a properly edited /etc/pam.d/system-auth file taken from CentOS 4.4 (some lines have been wrapped for readability; do not wrap them when editing the file):
    #%PAM-1.0
    # This file is auto-generated.
    # User changes will be destroyed the next time authconfig is run.
    auth     required   /lib/security/$ISA/pam_env.so
    auth     sufficient /lib/security/$ISA/pam_unix.so likeauth nullok
    auth     sufficient /lib/security/$ISA/pam_krb5.so
    auth     required   /lib/security/$ISA/pam_deny.so
    
    account  sufficient /lib/security/$ISA/pam_unix.so
    account  sufficient /lib/security/$ISA/pam_krb5.so
    account  sufficient /lib/security/$ISA/pam_succeed_if.so
        uid < 100 quiet
    account  required   /lib/security/$ISA/pam_deny.so
    
    password requisite  /lib/security/$ISA/pam_cracklib.so retry=3
    password sufficient /lib/security/$ISA/pam_unix.so nullok
        use_authtok md5 shadow
    password  required  /lib/security/$ISA/pam_deny.so
    
    session  required   /lib/security/$ISA/pam_limits.so
    session  required   /lib/security/$ISA/pam_unix.so
  7. Edit the /etc/nsswitch.conf file to include “ldap” as a lookup source for passwd, shadow, and groups.

At this point we are now ready to test our configuration and, if successful, to perform the final step: to join the Linux server to Active Directory for authentication.

Test the Configuration

To test the Kerberos authentication, use the kinit command, as in kinit <AD username>@<AD domain DNS name>; this should return no errors. A klist at that point should then show that you have retrieved a TGT (ticket granting ticket) from the AD domain controller. If this fails, go back and troubleshoot the Kerberos configuration. In particular, if you are seeing references to failed TGT validation, check to make sure that both your Linux servers and AD domain controllers have reverse lookup (PTR) records in DNS and that the Linux server’s /etc/hosts file listed the FQDN of the server first instead of just the nodename.

<aside>Some readers and some other articles have suggested the use of the AD domain DNS name in the /etc/krb5.conf file instead of an AD domain controller specifically; I recommend against this. First, I believe it may contribute to TGT validation errors; second, it is possible to list multiple KDCs (AD DCs) in the configuration. Since the only major reason to use the AD domain DNS name instead of the DNS name of one or more DCs would be fault tolerance, then it doesn’t really gain anything.</aside>

To test the LDAP lookups, use the getent command, as in getent passwd <AD username>; this should return a listing of the account information from Active Directory. If this does not work, users will not be able to login, even if Kerberos is working fine. If you run into errors or failures here, go back and double-check the LDAP configuration. One common source of errors is the name of the LDAP bind account, so be sure that is correct.

Join the Linux Server to Active Directory

This is the final step. Don’t try this step until you’ve successfully tested the configuration. After this step is completed, you are finished and AD users will be able to login to Linux-based systems (assuming the AD users have been properly configured for Linux logins).

To join the Linux server to Active Directory, follow these steps:

  1. Verify the Samba configuration. Be sure the following settings are specified in /etc/samba/smb.conf:
    workgroup = <NetBIOS name of AD domain>
    security = ads
    realm = <DNS name of AD domain>
    use kerberos keytab = true
    password server = <Space-delimited list of AD DCs>
  2. Use kdestroy to destroy any existing Kerberos credentials you may have; then run kinit <Domain administrative account>@AD.DOMAIN.NAME to get a Kerberos ticket for an account that is a domain administrator account.
  3. Run net ads join to join the Linux server to Active Directory. This command will automatically create a computer object in Active Directory and add the appropriate SPNs (service principal names) to the computer object. In addition, it will populate the local Kerberos key table (/etc/krb5.keytab, by default) with the correct entries for authentication against Active Directory.

Only one small detail remains: how to deal with home directories for users logging into Linux systems.

Deal with Home Directories

Unlike Windows systems, home directories are required on Linux-based systems. As a result, we must provide home directories for each AD user that will log in to a Linux-based system. We basically have two options here:

  • Use the pam_mkhomedir.so PAM module to automatically create local home directories “on the fly” as users log in. To do this, you would add an entry for pam_mkhomedir.so in the session portion of the PAM configuration file.
  • Use the automounter to automatically mount home directories from a network server. This process is fairly complicated (too involved to include the information here), so I’ll refer you to this article on using NFS and automounts for home directories. This has the added benefit of providing a foundation for unified home directories across both Windows and Linux systems.

(There is a third option as well: manually create home directories before users can log in.)

Once you’ve settled on and implemented a system for dealing with home directories, you are finished! UNIX-enabled users in Active Directory can now login to Linux-based systems with their Active Directory username and password.

What’s not addressed in this article? Password management. In this configuration, users will most likely not be able to change their password from the Linux servers and have that change properly reflected in Active Directory. I’ll try to work on that for version 5 of the instructions.

I hope you find this information helpful. As always, feel free to post corrections, additions, or suggestions in the comments below.

Tags: , , , , , , , ,

« Older entries