blog.scottlowe.org

I just wanted to provide a quick update on some articles I have in the works to be (hopefully) published soon.

• I’m working on an article discussing when to use various NIC teaming configurations with VMware ESX. There are some significant repercussions here for a variety of network configurations, but especially so for configurations involving IP-based storage (iSCSI or NFS).
• I’m finally wrapping up an article on the Xsigo I/O Director. I’ve been working a Xsigo VP780 in the lab for quite some time, and this article will provide a brief overview along with some tips and tricks.
• I received word from HP that I should be getting a ProCurve switch in my lab soon, so that means I can provide a ProCurve-oriented version of this NIC teaming and VLAN trunking article.
• I have some notes on using NetApp Open Systems SnapVault (OSSV) in conjunction with VMware ESX that I plan to post here as well.

New versions of the Linux and Solaris AD integration articles are on the way as well, starting with an update of the Solaris instructions to accommodate Solaris 10 Update 5 and Windows Server 2008.

If there’s anything else you’re interested in seeing, let me know in the comments. Thanks for reading!

UPDATE: The NIC utilization article is available here.

I came across an interesting paper discussing how various virtualization environments protect well-behaved VMs from misbehaving VMs. The paper is available here.

In the tests described in the paper, researchers used virtual machines on Xen 3.0 (the open source hypervisor not the commercial XenServer product, as far as I can tell), VMware Workstation 5.5, and “Open Solaris 10” (quotes mine). As pointed out in the paper, these three environments represent paravirtualization, full virtualization, and OS virtualization (or containers). I’m not sure if the researchers actually meant OpenSolaris; I suspect not since that’s a very recent release. Instead, I believe they probably just meant Solaris 10. On Xen and VMware Workstation, both running under Linux, they used Linux-based VMs; on Solaris, they used additional instances of Solaris. Each VM or instance ran Apache 2 and was tested using physical clients to connect to the HTTP server in each VM.

The results are interesting; VMware showed the best protection of well-behaved VMs from a misbehaving VM, followed by Xen with Solaris Containers providing the least protection. The level of protection was tested using a memory consumption stress test, a CPU stress test, a disk I/O stress test, and a network I/O stress test. I’d encourage you to have a look at the full paper for all the details.

These results are very interesting, but I wonder how much the results would change if we were to use VMware’s ESX server product line instead of one of the hosted products like VMware Workstation? As a product representative of “full virtualization” solutions, I’d be curious to know if the results seen with VMware Workstation were also seen with ESX.

In any case, the results are a validation of what we, as consultants, have been talking about: full virtualization provides the best isolation of well-behaved workloads from ill-behaved workloads, preventing a workload in one VM from affecting other workloads due to mishandling of CPU, RAM, disk, or network resources. As the researchers conclude in the paper, “…it is clear that VMware completely protects the well-behaved VMs under all stress tests. Its performance is sometimes substantially lower for the misbehaving VM, but in a commercial hosting environment this would be exactly the right tradeoff to make.”

While at VMware Partner Exchange 2008 in San Diego this week, a few virtualization- or VMware-related headlines have popped in and caught my attention:

• Via Eric, I learned that vimsh has morphed into vmware-vim-cmd in version 3.5. Xtravirt’s updated document can be found here.
• Via Duncan, it looks as if a number of patches for ESX/ESXi 3.5 have been released. Time to put Update Manager through its paces…
• As several other bloggers have mentioned, VMware is now discussing in much greater detail the VMware Certified Design Expert (VCDX) certification. I suspect that the BC/DR and VI architectural workshops that are taking place at Partner Exchange this week—which incorporate a fairly intensive review and presentation process—are prepping professionals for the rigors they will have to endure to achieve VCDX. Bring it on!
• Sys-Con Media—which has republished a couple of my articles—published this interesting article from a KVM developer regarding the placement and architecture of I/O and I/O drivers in various virtualization solutions. Of course, he feels that KVM is the best, but that’s not necessarily surprising.
• Author David Davis has published a brief blog entry at SearchVMware.com that summarizes the use of NIC teaming and load balancing with VMware ESX Server. This blog post is particularly useful since it references some of my own content.

In a future post, I’ll probably delve into more detail an interesting and thought-provoking article from DCS titled “Microsoft Unveils GSNW 2.0″. It’s an interesting take on the (possible) repetition of history. In the meantime, I’d love to hear other people’s thoughts on this article—go read it, then come back here and add your thoughts in the comments below.

Reader Scott Merrill pointed out something to me in an e-mail regarding a Registry change that might be necessary in some Active Directory integration scenarios:

Finally, I would like to share one registry change that we’ve found to be necessary in our AD integration. By default, the MS LDAP server only returns 1,000 results. As a university department with more than 1000 active students, this limitation has caused us some frustration.

This KB article shows how to increase the number of results returned in a query: http://support.microsoft.com/kb/315071

We recently set MaxPageSize to 5,000. I don’t know if this will
introduce additional problems down the road, but at least it lets me fully enumerate all our AD users from a Linux machine with `getent passwd`.

If you have an Active Directory domain with more than 1,000 users in the DN specified in your LDAP configuration, then this is a Registry change you’ll want to investigate. Otherwise, you could find that your UNIX/Linux servers aren’t able to fully enumerate all the users in the domain.

Thanks, Scott!

The recently announced beta for VMware Workstation 6.5 includes support for Unity, a technology that was originally introduced with VMware Fusion. Unity allows for seamless windowing of VMs; that is, windows from VMs and windows from the host can be seamlessly intermingled so that the distinction between the virtualized OS and the host OS almost disappears. This is the continued emergence of the application agnosticism I discussed near the end of 2006 and furthers the possibility of collections of VMs becoming our new working environments.

It’s very cool technology, and it’s also very exciting to see VMware bringing this technology to VMware Workstation. I hope that VMware also takes some of the new Unity features—like the badge that helps identify which windows belong to a VM and to which VM they belong—and backport that to future versions of Fusion as well. Of course, I’d love to see all of Workstation’s functionality ported over to Fusion, but that’s just me.

There’s more information on Workstation’s new Unity functionality from Christian Hammond, a VMware developer, on his blog.

Reader Jeffrey Spear contacted me a while back with some problems he was experiencing in trying to integrate some Linux systems into Active Directory. Basically, Kerberos was working but LDAP wasn’t. He was able to use “kinit <AD username>” to generate a Kerberos ticket, but using the “getent passwd <AD username>” was not working. No error messages, nothing; it just didn’t work.

We traded e-mails back and forth for a while, and eventually he found the solution himself:

We work with a locked down version of OSs and in this case a domain policy on the Windows server was preventing the RHEL machines from accessing account info.  The policy was “Domain controller: LDAP server signing requirements” which was set to “Require signature.”  When I changed this setting to “None” it worked great.

This is good information and important to keep in mind; I’ll be sure to incorporate this into the next revision of the Linux-AD integration instructions. (No, I don’t have a timeframe on when that will be!)

In the meantime, if anyone has a workaround for this problem that will allow LDAP to work with signatures enabled or required, I’d love to hear it. Speak up in the comments below!

Back on March 5 blogger extraordinaire Alessandro Perilli of virtualization.info revealed that Cisco had chosen KVM as the virtualization platform for IOS-XE, the new Linux-based version of IOS that runs on the recently introduced ASR series of routers.

If you were like me, you may have been wondering exactly how Cisco was putting KVM to use. No need to wonder any longer! Colleague and fellow blogger Colin McNamara has written up a detailed and in-depth discussion of the ASR1000 and how it uses KVM to provide virtualized instances of IOS-XE. Colin also discusses the role of the QuantumFlow processor and, believe it or not, the role of Popeye and Spinach. (Go read the article. It will make sense when you’re done.) Nice work, Colin!

I just finished reading the book titled Xen Virtualization: From Technologies to Solutions, published by Packt Publishing and written by Prabhakar Chaganti.  Overall, the book was helpful in getting up to speed with Xen, although it appears that English was not the native language for either Mr. Chaganti, his editor, or both.

The book is subtitled “A fast and practical guide to supporting multiple operating systems with the Xen hypervisor,” and it does live up to that subtitle.  The book very quickly moves into some hands-on exercises using a Linux host and the open source Xen hypervisor.  The exercises are fairly pertinent to the topic being discussed, and I especially liked the “What just happened?” sections after each hands-on procedure.  In those sections, the author breaks down the steps, the intended results, and the reasoning behind the procedure.  In my view, that’s a very helpful way to build understanding of the product.

My only complaint is—as I mentioned earlier—that English appears not to have been the native language for the author and/or editors.  The wording sometimes gets in the way of the content, making it more difficult than it should be to understand what the author is trying to say.  I would also say that I don’t think the book is worth the $40USD price tag that marked on the back.  At only 127 pages, $40 seems a bit steep.

Those issues aside, I found the book to be helpful in understanding Xen and some of Xen’s concepts.  I wouldn’t necessarily recommend this book to people who are both new to virtualization as well as new to Linux, as the material assumes a certain level of knowledge and experience with Linux.  Otherwise, if you have some Linux experience and want to get started with Xen, this book would be a good place to start.  (Just try to find the book on sale.)

A reader kindly shared with me some tips and tweaks he used to help resolve some performance issues with Linux-AD integration, and now I’d like to share them with you:

I ran into some nagging performance issues with Linux/AD integration the other day, and managed to solve them (mostly)—I thought you might be interested in the solution.

Since I’m not exactly following your integration guide (I use DNS lookups to locate LDAP servers in AD, and use GSSAPI authentication for nss_ldap instead of a binddn), I have a bit more overhead on my getent system calls, that was starting to get noticable when it came to getting directory listings, etc.

Two things I have done to alleviate this issue:

1. Since we have a flat catalog, and all of our DCs are also GCs, I
   disabled recursion. This is not universally relevant, but it does assist me
2. I edited /etc/nscd.conf to set a 5 minute cache time for passwd and group (but not host!) entries. Data is still usually piping fresh, but now we only call nss_ldap once every 5 minutes, instead of every time we need to know who owns a file. Then I started nscd, and set it to start on boot.

NSCD is a standard part of most RHEL and derivative installs.

Since implementing the above, the user-level experience is no longer
discernably different than the old /etc/passwd method of authentication/authorization.

This is good information to have.  Thanks to Brandon for sharing his experience!

I’ve received some feedback from a reader who alerted me to some sort of interaction between the Local Security Policy on the Windows side and Linux servers authenticating to Active Directory via Kerberos/LDAP/Samba.  I haven’t quite been able to get to the root issue yet, but here’s the high level overview.

The reader was seeing strange delays at the end of a Linux logon process that seemingly could not be explained.  After jumping through all the hoops, another administrator within the organization changed the Local Security Policy setting that governed the use of LM and NTLM authentication, and the delays disappeared.

The policy had been set to allow both LM and NTLM authentication; when changed to allow only NTLM authentication, the delays disappeared immediately.  The Linux server in question did have Samba installed, so apparently Samba was timing out trying the LM authentication; this caused the delays.  Of course, this is all just speculation, as we don’t know exactly why the policy change eliminated the delay.

In any case, since I’ve been pushing the use of Samba in my latest integration instructions (Solaris version here), I thought it might be prudent to mention this feedback.  In the event you start seeing some strange delays in your Linux authentication requests, check the Local Security Policy and see if LM authentication is being permitted.  That might just be your culprit.