June 2008

The VMware peformance team blog (found here) published an entry last week on ESX scheduler support for multiprocessor VMs. The blog entry itself was quite informative; in particular, the information about ESX 3.x’s “relaxed co-scheduling” mechanism was very useful.

Long-time users of ESX will recall that mixing uniprocessor (UP) and multiprocessor (MP) workloads on an ESX server was very strongly discouraged, especially in systems with a lower number of cores. This was due to the “strict co-scheduling” mechanism used in ESX 2.x. The “relaxed co-scheduling” mechanism introduced in ESX 3.x that allows for far greater flexiblity in scheduling UP and MP VMs. To quote the blog entry:

Relaxed co-scheduling significantly reduces the possibility of co-scheduling fragmentation, improving overall processor utilization.

I was not aware that the vCPU scheduling mechanisms had changed between 2.x and 3.x, so this is good news.

However, the real treasure in this blog entry—for me, anyway—was the link to the in-depth performance documents being posted in the Performance Community Forum. Here are some very detailed documents that provide outstanding information on topics like:

• the VMkernel scheduler
• how ESX co-schedules SMP VMs (more detailed than the blog entry)
• using (or not using) hyper-threading with ESX

Overall, this is a ton of good content which will be very helpful to anyone seeking to solidify their knowledge of the ESX platform.

Like everyone else in the virtualization world (except for perhaps the folks in Palo Alto, CA), there’s a lot of Hyper-V stuff crossing in front of me.

This time it’s an article on storage options for Hyper-V, written by Jose Barreto. (You’ll recall that I referenced Jose’s clustering article a few days ago.) Out of the wide variety of blogs coming out of Microsoft, Jose’s is one that I have really, truly found informative and helpful. The home page for his blog is here.

Jose also wrote a follow-up article on Hyper-V’s storage options where he discussed booting from iSCSI.

Great work, Jose! Keep it coming.

Here in Virtualization Short Take #11, I offer to you a collection of virtualization-related news and tidbits and my thoughts on them.

• I seem to be on a bit of kick reading Ryan Arneson’s stuff these days. This time is actually an older post of Ryan’s on using the COMSTAR stuff from Sun with ESX. It’s an interesting read. I’m quite fascinated by the myriad things that Sun is doing with storage, and I hope that some of these actually get backed with good execution. I’ve guess I’ve heard the saying “Sun is where storage goes to die” from too many Sun veterans.
• I was notified of this post by Chris Barclay of Virtual Iron regarding a comparison of Virtual Iron Virtualization Manager and Citrix XenCenter. This is an interesting comparison considering that both products are built on the same underlying hypervisor (Xen). In this case, Chris makes the argument that management is the piece that sets one virtualization solution apart from other solutions, and that in this particular case Virtual Iron’s management capabilities far exceeds those provided by XenCenter. I don’t have any direct experience with either of these products, so I can’t attest as to the accuracy of his claims. While I don’t necessarily agree that the hypervisor is being commoditized, I do agree that management is increasingly becoming the factor that distinguishes solutions. In this regard Microsoft has an early lead, in my opinion, with cross-platform VM management inside Virtual Machine Manager 2008. Will other vendors follow suit?
• Last week the new VMware Networking blog posted a notice about a new whitepaper jointly authored by VMware and Cisco. Duncan over at Yellow Bricks also picked this up, but from a different source; the whitepaper, however, appears to be the same from both sources. I haven’t had the opportunity to fully review it yet, but I do plan to do so and will highlight any notable recommendations here.
• Chad Sakac, the “VMware Guru” for EMC, published an entry on stretched ESX clusters. This article was picked up by a number of other bloggers (here or here, for example), so I won’t rehash it all here again. The timing on the article was helpful; he wrote that and not more than two days later I had a customer asking about doing this very thing. Personally, I agree with Chad that it’s generally a bad idea, and so it was handy to be able to point the customer to this article as further support. One other thing I did get out of Chad’s post—how many of you picked up that up to 10 different isolation addresses can be configured? Is that in the documentation somewhere and I just missed it?
• Continuing on with Chad, it appears that an old VMware HA article of mine is useful in helping to understand how the VMware HA admittance algorithm works. Chad’s article provides excellent details on the key concepts to understand.
• Most readers have probably seen the article describing how to access the ESXi command line. This article also shows you how to enable SSH access to that CLI. I found this information so handy that I added it to my del.icio.us bookmarks. As ESXi gains broader adoption, this kind of stuff will be very useful.
• With the release of Hyper-V, comparisons of Hyper-V vs. ESX will become much, much more common. Here’s another one for review as well. I’ll echo the comments in this article regarding the comparisons: it’s not about the brand, or the technology, it’s about the solution.
• I’ll have to partially disagree with the sentiment behind this article regarding the use of virtualization as a DR tool. The article intends to present 5 things that should be considered when using virtualization for DR, but does not, IMHO, accurately present some of the challenges around virtualization for DR. How are the VMs being replicated over to the DR site? Replication technologies need to be properly coordinated with the virtualization software so that the data being replicated is consistent and useable. If this is synchronous replication it’s not as much of an issue, but it’s definitely an issue with asynchronous replication. What about registering VMs on the DR site? How does one handle VirtualCenter in this kind of scenario? Is testing failover really that easy? My experience indicates that while virtualization can certainly assist in creating a good DR plan, it’s only one part of an overall DR solution, and it can create its own unique challenges. Again, the timing of this is interesting; I just came across the article after finishing up a presentation about the use of virtualization in disaster recovery solutions.
• Anyone working in the VDI environment has almost certainly had more than their fair share of discussions about remote display protocols. This article on x86virtualization.com provides a decent overview of VNC, RDP, ICA, and Net2Display. Seems like I recall seeing something somewhere about VMware assisting in the development of Net2Display; anyone know anything more about that?

I guess that about does it for this round. Thanks for reading, and feel free to share your thoughts in the comments.

Today Microsoft released Hyper-V 1.0, their bare-metal hypervisor. I haven’t been able to find much information on the Internet about it yet other than Alessandro’s announcement and this article at Network World; there was nothing on the Microsoft web site about it at the time of this writing.

I’ve covered Hyper-V extensively in the last few weeks due to my Tech-Ed coverage, so I’ll refer readers back to some of the Hyper-V related sessions for more information on Hyper-V architecture, networking, security, or storage:

VIR367: Hyper-V Security and Best Practices
VIR250: Advanced Storage Connectivity for VMs
VIR358: Hyper-V Architecture, Scenarios, and Networking
Significant Networking Problem with Hyper-V
Hyper-V Clustering Scenarios
More on Hyper-V and NIC Teaming

The release of Hyper-V now opens the door for Microsoft’s System Center team to finalize Virtual Machine Manager 2008, which I currently see as a notable competitive advantage over VMware due to VMM’s ability to manage both Hyper-V as well as VI3. There are a number of VMM 2008 posts available with more information also:

VIR253: Microsoft System Center VMM 2008, Part 1 of 2
VIR360: Microsoft System Center VMM 2008, Part 2 of 2
VIR350: System Center VMM Advanced Integration

The next few weeks will be interesting to watch; in particular, I’m interested to see what actions VMware will take to counter this new competitive threat.

UPDATE: Microsoft finally provided some URLs to their announcements:

Microsoft’s Hypervisor Technology Gives Customers Combined Benefits of Windows Server 2008 and Virtualization
It’s here! Windows Server 2008 Hyper-V is available for download…

I suppose it was bound to happen at some point.

I’ve been confused for the other Scott Lowe. Yes, that’s right; there’s more than one.

The other Scott Lowe writes for TechRepublic.com, and currently works for either Elmira College or Westminster College; I’m not sure which. (I’ve seen both; I think it’s Westminster.) He’s published a couple of books, including one on Exchange Server 2007 that was published by Microsoft Press, and owns the slowe.com domain.

Normally, being mistaken for the other Scott Lowe wouldn’t be a problem; he seems like a smart, well-respected, knowledgeable guy. In this case, though, it was some business contacts at NetApp that picked up this article in which Scott Lowe purchased an EMC storage array. Clearly you can see where that would cause some confusion.

So, just to set the record straight: I didn’t purchase an EMC storage array. Nor did I purchase a Dell M1000e blade chassis. I also don’t know anything about open source IP telephony. Sorry. Not that there’s anything necessarily wrong with any of these products; it just wasn’t me. I do have a copy of the aforementioned Exchange Server 2007 book; but it’s not my name on the front cover, unfortunately.

I hope this helps to clear things up. However, if you’ve accidentally mistaken me for him, and you’d rather read his stuff, this TechRepublic.com search should help you out.

Now, back to looking at my Toshiba CIX1200 phone system. Oh, wait—that wasn’t me, either…

I ran into a recent issue with a customer who was having problems getting VLANs to work as expected with ESX. The basic scenario was that ESX would refuse to work properly with a VLAN that was marked as the native (or untagged) VLAN. This was causing no end of grief for this customer.

I’ve discussed VLANs extensively—first with this blog post, then again here, and again in this SearchVMware.com tip—so I was confident that I could help the customer resolve this issue. Granted, the customer was using Nortel switches, with which I am completely unfamiliar, but a switch is a switch, right?

Not quite. While the configuration seemed correct in all ways, it turns out there is a checkbox somewhere labeled “untag-default-vlan”. If this box is not checked, then the default VLAN gets tagged. Since ESX wasn’t configured with a VLAN tag, then it doesn’t see the network traffic. Once that box gets checked, then the default (or native) VLAN doesn’t get tagged and will be properly recognized by an ESX port group without a VLAN tag configured.

So, if you’re using Nortel switches and having problems with VLANs, double-check this setting.

My original article on Hyper-V’s issues with NIC teaming has gotten a fair amount of attention.

First Keith Ward over at Virtualization Review blogged about this issue. In his initial post, Keith basically pointed out the issue and then asked the readers for feedback: is this really as big of an issue as it seemed? The readers who responded were split; one blasted Hyper-V and the other wasn’t too concerned.

Keith followed that up with another post in which he provides a response from Microsoft regarding this issue:

NIC Teaming is a capability provided by our hardware partners such Intel and Broadcom. Microsoft supports our partners who provide this capability. This is true whether the customer is running Windows, Exchange, SQL, Hyper-V, etc. We’ll have a detailed KB article about this coming out soon.

Keith’s second article was then also picked up by DABCC.

While Microsoft is sticking to the “this is a device driver issue” mantra, I’m not so sure I agree. I can see their position to a point. In Keith’s second post, analyst Chris Wolf brings up storage drivers. This is similar in that Microsoft relies upon the storage vendors to provide device-specific modules (DSMs) that provide the multipathing functionality. So, like with the NIC teaming, Microsoft is pushing the functionality back to the device drivers and vendors who write them.

But that’s as far as this comparison can be taken. Microsoft officially supports storage multipathing; they don’t officially support NIC teaming. (See this KB article or this KB article.) In addition, Microsoft provides an official framework in which the storage vendors can operate: the MPIO framework. There is no such framework for network redundancy. In fact, if such a framework existed then much of the dissatisfaction with Microsoft over this issue would be alleviated, in my opinion.

Instead, there is no framework to provide official NIC redundancy for any Microsoft product running on Windows Server, and Windows itself doesn’t provide that functionality. Users are forced to adopt unsupported means to provide NIC redundancy. Why shouldn’t they be upset?

By the way, since publishing the first article I’ve been contacted by one of the presenters of the VIR358 session during this which issue came to light, but he has not yet been able to provide any additional information. As soon as more information is available, I’ll be sure to let everyone know here.

After my coverage of Tech-Ed 2008, I think a greater number of Microsoft people are reading my blog. One of them, Jose Barreto, sent me an e-mail to notify me about a post he’d written on the various ways in which to implement Windows Server Failover Clustering with Hyper-V. Quoting from his post:

There are many ways to implement Windows Server Failover Clustering with Hyper-V. I could actually find five unique methods to do it. Some of them will actually not give you a fully fault-tolerant solution, but most of them actually make sense in specific scenarios (even if only for demonstrations). In any case, just trying to understand and differentiate them will probably be a good exercise.

It’s a good read, and I recommend reading and reviewing it if you need to brush on up on how to combine clustering and Hyper-V.

Thanks for the heads-up, Jose, and thanks for the well-written article. Keep up the good work!

In UNIX/Linux integration scenarios, it’s useful to know which accounts have been UNIX-enabled, i.e., have had the UID number, NIS domain, login shell, and home directory attributes configured.

It’s certainly very possible to do this with command-line tools such as AdFind or DsQuery, but users may also find it useful to have a saved query available within the Active Directory Users & Computers console for easy reference.

The way to do this is define a custom query using this string:

(objectCategory=Person)(objectClass=User)(uidNumber=*)

If you add just this text and nothing else in the “Find Custom Search” dialog box (the Advanced tab), then the console will automatically add ampersands and additional parentheses to turn it into a “proper” LDAP query that will show you any account that has a UID number configured. Certainly, additional fields like loginShell or unixHomeDirectory could be added as well, but this query will probably be sufficient for most instances.

I started not to publish this, but figured if I couldn’t remember the exact syntax then someone else might not be able to remember the syntax either. This one is as much for me as it is for others.

A lot of virtualization-related articles were published while I was in Orlando at Microsoft Tech-Ed 2008. Here are a few that caught my attention over that time period as well as in the last few days.

• Rich Brambley over at VM /ETC is on a roll with a couple of good articles, one on VMFS storage sizing for performance and one on downgrading the VM HAL after P2V. The title for that second article was “How to P2V Multi-processor servers to Uni-processor VMs”, and I was hoping that it was a new trick with VMware Converter or some other P2V tool that would actually take care of the HAL for me. Alas, not this time. Rich’s article is useful information nevertheless, don’t get me wrong. I am curious, though, as to the source of Rich’s storage sizing recommendations. Rich, can you share where you derived those recommendations?
• Ryan Arneson shares some of his experiences in using an X4500 with ZFS as an NFS datastore for ESX. Ryan’s post reminds readers of the some of the requirements for using NFS with ESX, so readers new to that sort of configuration may find it helpful.
• Gabe brings us, courtesy of this VMware Communities thread, some information on getting DRS VMotion information from the VC database. That’s handy. In an earlier article, Gabe also weighed in on storage sizing as well. This seems to be getting quite a bit of attention recently (gee, I can’t imagine why). Readers, I’d love to hear your thoughts on storage sizing approaches, algorithms, etc. Please share them in the comments!
• I found this article discussing NFS vs. CIFS for VMware. At first I was a bit baffled, but then I realized the author must have been talking about VMware’s hosted products like VMware Server not ESX. Anyone else tried this?
• Rick Blythe aka VMwareWolf posted an article about VM customization failing after VC 2.5 upgrade. Fortunately, the fix is easy; just download and install the correct version of the Sysprep tools and put them in the right location on the VC server. The details are all provided at his site, so check that out.
• The VMware Fusion Team announced support for running Mac OS X Leopard Server in Fusion 2.0 back during the WWDC, but it seems that Parallels may have beaten them to the punch with an actually shipping product.
• Schley Andrew Kutz has released a version of the VI Toolkit for .NET that supports mocking. More information on mocking the VI Toolkit is available here. I haven’t seen any sample taunts yet. (You’ll get that in a few minutes.)

In addition, as a follow-up to my Tech-Ed coverage of the Microsoft Assessment and Planning (MAP) toolkit, I found these two articles in my list of “things to blog about”:

Microsoft Assessment and Planning How-To Series: Part 1 (Server Virtualization Candidacy Reporting)
[VIDEO] Microsoft Assessment and Planning How-To Series: Part 1 (Server Virtualization Candidacy Reporting)

Note also that the beta program for MAP 3.1 is already running; this version will add support for Hyper-V. More information here. (I couldn’t remember if I stated anything about this my Tech-Ed session liveblogs.)

That about does it this time around. Thanks for reading!