January 2010

You are currently browsing the monthly archive for January 2010.

EMC Celerra Optimizations for VMware on NFS

Sunday, January 31, 2010 in Storage, Virtualization by slowe | 19 comments

Recently Jason Boche posted some storage performance numbers from his EMC Celerra NS-120. Out of those storage performance tests, Jason noted that the NFS performance of the NS-120 seemed a bit off, so he contacted EMC’s vSpecialist team (also known as “Chad’s Army”, but I like vSpecialists better) to see if there was something that could or should be done to improve NFS performance on the NS-120. After collaborating internally, one of our team members (a great guy named Kevin Zawodzinski) responded to Jason with some suggestions. I wanted to reproduce those suggestions here for everyone’s benefit.

Note that some of these recommendations are already found in the multi-vendor NFS post available on here on Chad’s site as well as here on Vaughn Stewart’s site.

In addition, most if not all of these recommendations are also found in the VMware on Celerra best practices document available from EMC’s web site here.

Without further ado, then…

  • As has been stated on multiple occasions and by multiple people, be sure that virtual machine disk/application partitions have been properly aligned. We recommend a 1MB boundary. Note that Windows Server 2008 aligns at a 1MB boundary automatically.
  • Use a block size of 8KB unless other recommended or required by the application vendor. Note that the default NTFS block size is 4KB. (Pages 128 through 138 of the Celerra best practices document contain more information on this bullet as well as the previous bullet.)
  • Turn on the uncached write mechanism for NFS file systems used as VMware datastores. This can have a significant performance improvement for VMDKs on NFS but isn’t the default setting. From the Control Station, you can use this command to turn on the uncached write mechanism:
    server_mount <data mover name> -option <options>,uncached <file system name> <mount point>
    Be sure to review pages 99 through 101 of the VMware on Celerra best practices document for more information on the uncached write mechanism and any considerations for its use.
  • Change the VMware ESX settings NFS.SendBufferSize and NFS.ReadBufferSize to a value that is a multiple of 32. The recommended value is 64. See page 73 of the best practices document for more details.
  • If you’ve adjusted the NFS.MaxVolumes parameter in order to have access to more than 8 NFS datastores, you should also adjust Net.TcpIpHeapSize and Net.TcpIpHeapMax parameters. The increase should be proportional; if you increase the maximum volumes to 32 (a common configuration), then you should increase the other parameters by a factor of 4 as well. Page 73 of the best practices document covers this. This VMware KB article and this VMware KB article also have more information.
  • Although not directly related to performance, best practices call for setting NFS.HeartbeatFrequency (or NFS.HeartbeatDelta in VMware vSphere) to 12, NFS.HeartbeatTimeout to 5, and NFS.HeartbeatMaxFailures to 10.
  • Ensure that the LUNs backing the NFS file systems are allocated to the clar_r5_performance pool. This configuration will balance the load across the SPs, LUNs, etc., and help improve performance.

Depending upon the other workloads on the system, another NFS performance optimization is to ensure that the maximum amount of write cache on the SPs is configured. However, be aware this may impact other workloads on the array.

As Jason noted in his post, implementing these changes—especially the uncached write mechanism—offered performance benefits for NFS workloads.

Keep these configuration recommendations in mind when setting up your EMC Celerra for VMware on NFS.

Tags: , , , , ,

Virtualization Short Take #34

Sunday, January 24, 2010 in Virtualization by slowe | 6 comments

Welcome to Virtualization Short Take #34, my occasionally-weekly collection of virtualization-related links, posts, and comments. As usual, this information is a hodge-podge of information I’ve gathered from across the Internet over the last few weeks. I hope that you find something useful or helpful here, and thanks for reading!

  • First up is Arne Fokkema’s PowerCLI script to check Windows VM partition alignment. As one commenter pointed out, the fact that the starting offset isn’t 65536—which is what Arne’s script checks—doesn’t necessarily mean that it isn’t aligned. Generally, you can align a Windows partition by setting the starting offset to any number that is evenly divisible by 4096 (4K). If I’m not mistaken, setting the partition offset to 65536 (64K) also ensures that the partition is stripe-aligned on EMC arrays.
  • Here’s a useful reminder to be sure to keep your dependencies in mind when designing VMware vSphere 4 environments. If you design your environment to rely upon DNS—a common situation, since VMware HA is particularly sensitive to name resolution—then be sure to appropriately architect the DNS infrastructure. This “circular dependency” is one reason why I personally tend to keep vCenter Server on a physical system. Otherwise, you have the virtualization management solution running on the infrastructure it is responsible for managing. (Yes, I know that it’s fully supported for it to be virtualized and such.)
  • Forbes Guthrie’s article on incorporating Active Directory authentication and sudo into the kickstart process is a good read. With regard to his note about enabling root SSH access because of an inability to access the Active Directory DCs: I know that in ESX 3.x you could still log in at the Emergency Console when Active Directory connectivity was unavailable; does anyone know if this is still the case with ESX 4.0? I haven’t taken the time to test it yet.
  • Oh, and speaking of Active Directory authentication, Forbes also published this note about Likewise AD authentication supposedly included in ESX 4.1. Looks like someone at Likewise accidentally spilled the beans…
  • I’m sure that everyone has seen the article by Duncan about the ESX 3.x bug that prevents NIC teaming load balancing from working on the global vSwitch configuration, but if you haven’t—well, now you have. Here’s the corresponding KB article, also linked from Duncan’s article. Duncan also recently published a note about an error while installing vCenter Server that is related to permissions; read it here.
  • Are there even better days ahead for virtualization and those involved in virtualization? David Greenfield of Network Computing seems to think so. The comments in the article do seem to bear out my statements that virtualization experts now need to move beyond consolidation and start helping customers tackle the Tier 1, high-end applications. I believe that this is going to require more planning, more expertise, and more knowledge of the applications’ behaviors in order to be successful.
  • Stephen Dion of virtuBLOG brings up a compatibility issue with Intel quad-port Gigabit Ethernet network adapters when used with VMware ESX 4.0 Update 1. Anyone have any updates or additional information on this issue?
  • If you’re considering virtualizing Exchange Server 2010 on VMware vSphere, be sure to read Kenneth’s article here about Exchange 2010 DAGs and VMotion. At least live migration isn’t supported on Hyper-V, either.
  • Want to run a VM inside a VM? This post on nested VMs over at the VMware Communities site has some very useful information.
  • Paul Fazzone (who I believe is a product manager for the Nexus 1000V) points out a good point-counterpoint article with Bob Plankers and David Davis that discusses the benefits and drawbacks of the Cisco Nexus 1000V. Both writers make excellent points; I guess the real conclusion is that both options offer value for different audiences. Some organizations will prefer the VMware vSwitch (or Distributed vSwitch); others will find value in the Cisco Nexus 1000V. Choice is a beautiful thing.
  • Jason Boche published some performance numbers for the EMC Celerra NS-120 that he’s recently added to his home “lab” (I use the term “lab” rather loosely here, considering the amount of equipment found there). Not surprisingly, Fibre Channel won out over software iSCSI and NFS, but Jason’s numbers showed a larger gap than many expected. I may have to repeat these tests myself in the EMC lab in RTP to see what sorts of results I see. If only I still had the NS-960 that I used to have at ePlus….sigh.
  • Joep Piscaer has a good post on Raw Device Mappings (RDMs) that definitely worth a read. He’s pulled together a good summary of information on RDMs, such as requirements, limitations, use cases, and frequently asked questions. Good job Joep!
  • Ivo Beerens has a pretty detailed post on multipathing best practices for VMware vSphere 4 with HP EVA storage. The recommendation is to use Round Robin with ALUA and to reduce the IOPS limit to 1. Ivo also presents a possible workaround to the IOPS “random value” bug that Chad Sakac discussed in this post some time ago.
  • Here’s yet another great diagram by Hany Michael, this time on ESX memory management and monitoring.
  • This post tells you how to modify your VMware Fusion configuration files to assign IP addresses for NAT-configured VMs. If you’re familiar with editing dhcpd.conf on a Linux system, the information found here on customizing Fusion should look quite familiar.
  • Back in 2007, I wrote a piece on using link state tracking in blade deployments. This post wasn’t necessarily virtualization focused, but certainly quite applicable to virtualization environments. Recently I saw this article pop up on using link state tracking with VMware ESX environments. It’s good to see more people recommending this functionality, which I feel is quite useful.
  • Congratulations to Mike Laverick of RTFM, who this past week announced that TechTarget is acquiring RTFM and its author, much like TechTarget acquired BrianMadden.com (and its author) last year. Is this a new trend for technical blog authors—build up a readership and then “sell it off” to a digital media company?

Here are some additional links that I stumbled across, but for which I haven’t yet fully assimilated or processed. You might see some more in-depth blog posts about these in the near future as they work their way through my consciousness.

Lab Experiment: Hypervisors (Virtualization Review)
The Backup Blog: Avamar and VMware Backup Revisited
VMware vSphere Capacity IQ Overview – I’m Impressed!

Well, that wraps it up for now. Thanks for reading and feel free to speak out in the comments below.

Tags: , , , , ,

Congrats to Hyper9

Tuesday, January 19, 2010 in Virtualization by slowe | No comments

I received an e-mail from David Marshall of Hyper9 (David also runs VMBlog.com) with some good news: Hyper9 has had their strongest quarter yet, despite the less-than-ideal economic conditions. Hyper9′s Virtual Environment Optimization (VEO) is a pretty well-regarded solution, and based on the limited hands-on time I’ve spent with it I like it. So, congratulations to Hyper9 for providing a bright spot in the economic landscape. Well done, guys!

The full press release is available here.

Disclosure: Hyper9 is a paid sponsor of this site.

Tags:

Thinking Out Loud: It’s Not Just the Technology

Tuesday, January 19, 2010 in Collaboration, Virtualization by slowe | 11 comments

I had a quick thought this morning while browsing this post by Lori MacVittie. She, in turn, was referring back to a post published on VMBlog.com about a virtualization prediction that 2010 would be the year that the network becomes fluid and virtual. (As a side note, the original article on VMBlog.com appears to be primarily a marketing exercise for a company that purports to help make the network fluid and virtual.)

Lori’s post, titled “A Fluid Network is the Result of Collaboration Not Virtualization,” clearly disagrees with the original VMBlog.com post and states that there’s more to creating a fluid network than just virtualization:

The network will become fluid—I absolutely agree—but that metamorphosis will not [happen] solely because of virtualization.

At first, I thought the post was about how organizations needed more than just technology to create an efficient, fluid, dynamic infrastructure; the title says “Collaboration Not Virtualization.” And it is—sort of. Really, Lori is focusing on the “collaboration of infrastructure through integration based on standards-based control planes.”

OK, I’ll agree with her that integration of infrastructure is required to bring about the fluidity that is the “Holy Grail” of data centers. But here’s my thought: what about the human factor? What about operations? What about processes and procedures? I’ve seen so many companies virtualize their infrastructure and fail to see the huge benefits they thought they would reap. Why? Because it was “status quo”: keep doing the backups the way you’ve always done it, keep patching the machines the way you’ve always done it, keep managing the OS instances—yes, you guessed it—the way you’ve always done it. Sure, virtualization is great in that you can keep these processes and procedures the same during and after consolidation through virtualization. But in my opinion, organizations will see a much larger impact if they pay close attention to the processes and procedures. By optimizing their processes and procedures for virtualization, organizations can take advantage of all that virtualization has to offer.

So what do you think? Do organizations really need to optimize their operations for virtualization? I’d love to hear your thoughts, so sound off in the comments. Thanks!

Tags: , ,

Resetting the Root Password on VMware ESX 4.0

Wednesday, January 13, 2010 in Linux, Virtualization by slowe | 11 comments

Earlier today, I had to reset the root password on a lab server running VMware ESX 4.0 Update 1. For some reason, the password we assigned yesterday when we built the server from scratch wasn’t working this morning. OK, no big deal, right? Just reboot the server into single user mode and away you go. I won’t bother to repeat the steps for getting into single user mode; go to this article and it will give you what you need (the article is written for ESX 3.5 but it works fine for ESX 4.0).

Because this is a lab environment we just wanted to assign a simple password that anyone on the team could easily remember. (I’m sure the security purists out there are screaming right now.) Unfortunately, once I had the ESX host booted into single user mode, the passwd command insisted on making me use a complex password. There didn’t seem to be any simple way around the restriction.

However, having spent a fair amount of time with PAM (Pluggable Authentication Modules) during my Linux-AD integration experiments, I figured there was a way around it by modifying the PAM configuration. Sure enough, the /etc/pam.d/system-auth-generic file contained a reference to pam_passwdqc.so, the library that is responsible for ensuring complex passwords. The fix, therefore, was to somehow remove pam_passwdqc.so from the PAM configuration so that I could assign a simple password.

The first thing I tried was simply commenting out the line for the module, but the passwd command then failed to work, reporting an error that the authentication token could not be obtained. Strike 1! Next, I leave the pam_passwdqc.so module commented out and try changing the next line to required instead of sufficient. Same error: strike 2!

Finally, I simply replaced the pam_passwdqc.so line with a reference to pam_cracklib.so (after making a backup copy of the original /etc/pam.d/system-auth-generic file, of course—it never hurts to be prepared). Success! I was able to assign a simple password to the lab server.

After putting the original /etc/pam.d/system-auth-generic back in place and rebooting the host, we were back in action! So, what was the lesson learned? You can’t stop someone who’s determined to get around security requirements! No, I’m just kidding…there is no lesson learned. I just thought someone might find this information useful or interesting. Enjoy!

Tags: , , , ,

A Couple GeekTool Scripts

Monday, January 11, 2010 in Macintosh, UNIX by slowe | 2 comments

I’ve been experimenting with GeekTool, a nifty Mac OS X Preference Pane that allows you to display information on your desktop. This information can be static text, images, or the output of a script. The last option is the most useful one, in my opinion, and that’s where I’ve been putting GeekTool to use for me. This isn’t going to be some long post on how to use GeekTool or why you should install it; rather, I just wanted to share a couple of short scripts that I wrote that you might find useful.

I use Mac OS X’s network location support extensively. I have separate locations for home (where I have a proxy server) and when I’m out and about (where there generally is no proxy server). So it’s important for me to be able to tell, quickly, which location is active. If the wrong location is active, then network connectivity is impaired.

To help, I use this command with GeekTool to display the network location on my desktop:

echo "Location: `networksetup -getcurrentlocation 2>&1 | tail -n 1`"

Note that if you are running as an administrative user on your Mac (which I don’t in order to reduce potential security risks), then the networksetup command I use above will probably behave differently for you. Since I’m not running as an administrative user, networksetup would throw an error at the command line. Thus why I had to redirect STDERR to STDOUT and filter it using tail. Now, a quick F11 to show the desktop and I can immediately see which network location is active.

I also recently added a script to show me what proxy servers are currently active. This is in anticipation of starting my new job at EMC. I don’t know if they have proxy servers on their network, but in the event they do I thought this next command might be handy:

echo "HTTP Proxy: `scutil --proxy | grep HTTP | sort | sed -n '3,3p' | awk '{print $3}'`"

This command displays the HTTP proxy host configured in your network settings. So, again, a quick F11 allows me to see which proxy hosts are configured and active on my Mac.

I actually wrapped several of these commands together into a shell script that you can download here if you’d like. I’m sure there is probably some bash black magic that could produce this output in a more efficient way; feel free to post suggestions for improvement if you have any!

Of course, I also have a few other scripts running with GeekTool—one that displays system information, one that produces IP addresses and Airport (wireless LAN) information, etc.—but these two are probably most useful to me so far.

Tags: , ,

Virtualization Short Take #33

Thursday, January 7, 2010 in Virtualization by slowe | 3 comments

Welcome to Virtualization Short Take #33, the first installation of the Virtualization Short Take series for 2010! This installation will be a bit lean, but I hope that you find something useful among these nuggets of information.

  • This article by Kenneth Van Ditmarsch, backed up by this post by Chad Sakac, underscore the need for proper operational documentation for your virtualization environment. Organizations that have taken the time to prepare operational procedures and train their staff on using the documented procedures will, in my opinion, be far less likely to fall victim to this vSphere storage bug. I’m not saying you’ve got to go crazy on documentation, but take the time to document and validate the core procedures your team is using. I think you’ll find the results beneficial.
  • Speaking of vSphere bugs, Chad also describes a bug affecting vSphere 4 (including Update 1) involving NMP and Round Robin. If you change the I/O Operation Limit for Round Robin (using the esxcli command), you might find that the value gets changed to some random value upon reboot. The workaround is to not modify the I/O Operation Limit (the default value is 1000).
  • Scott Drummonds of VMware has been publishing a great series of articles on host swapping and memory overcommit. The series starts with a discussion on host swapping and the fact that VMware ESX does not track working sets within every VM (it would be too much overhead). He continues with this post on using SSDs to help alleviate potential host swapping performance concerns (also see this article that Scott references in his post). In the last two posts, Scott debunks some misconceptions about memory management and then goes to show why memory overcommit is important in optimizing memory utilization. Definitely some good stuff.
  • The VMware Communities blog post about using SSDs to improve performance when memory is overcommitted (found here) put me to thinking. In the tests documented in that post, local SSDs were used. What if EFDs were used instead? I’d be curious to know the results. This would support a boot from SAN approach that is more amenable to Cisco UCS model of stateless computing (although I’ve said before that I’m not entirely sold on stateless computing in a virtualized environment, since the hypervisor negates some of the benefits).
  • There are some areas where the Cisco UCS stateless computing model really shines; Steve Chambers describes one such use case in this post on multi-tenant DR with Cisco UCS.
  • Here’s a useful document on installing VMware ESXi on Cisco UCS using the UCS Manager KVM. Last time I tried the UCS Manager KVM on my Mac, it was barely usable and you couldn’t attach media; hopefully it’s improved since then.
  • Arnim van Lieshout has a great post on geographically dispersed VMware clusters. One thought that occurred to me as I was reading this post was that while Arnim’s post was written from the perspective of a production site and a DR site, the same challenges affect the use of external cloud providers and vCloud. As VMware and VMware’s partners start to address these challenges, not only does the idea of geographically dispersed clusters start to look more realistic and more flexible, but so too does the idea of leveraging additional capacity from a cloud provider via vCloud.
  • Interested in triggering an ESXi kernel panic on demand? Eric Sloof shows you how.
  • Finally, for users with the Nexus 1000V who want to update their ESX/ESXi hosts to Update 1 using the vihostupdate utility, Duncan’s post (and this associated VMware KB article) provides all the information necessary to make it work properly.

I did find a couple other useful posts that I haven’t had the time to properly read but which look interesting:

VMware Desktop Reference Architecture Workload Simulator (RAWC)
White Paper: VMware vSphere 4 Performance with Extreme I/O Workloads

That’s it for this time around. I welcome all courteous comments or thoughts on any of the links or posts I’ve mentioned here. Thanks for reading!

Tags: , , , , , ,

VMware, Zimbra, and Why I’m Not an Analyst

Tuesday, January 5, 2010 in Collaboration, Messaging, Virtualization by slowe | 9 comments

The rumors are swirling that VMware is going to buy Zimbra, an open source e-mail platform currently owned by Yahoo. You’ve probably all read the various news articles about the rumors and the events leading up to the rumored acquisition, so I won’t bother you with them again.

Yesterday on Twitter, I mentioned that the Zimbra acquisition didn’t quite make sense to me. I wasn’t the only one; several others mentioned it, too. That sparked a great discussion with Chris Wolf of the Burton Group (really sharp guy, by the way). His comments caused me to look at the purported acquisition in a different light where it starts to make a bit more sense. The discussion reminded me why I’m not an analyst: sometimes I don’t take a broad enough view. (It’s something I’ll work on improving.)

Most people look at the Zimbra acquisition (assuming that the rumor is accurate and it really does happen) and immediately recognize the intent to compete against Microsoft Exchange. However, if you look at the Zimbra acquisition strictly from the perspective of competing against Microsoft Exchange in the market as it currently exists, you’ll quickly come to the same conclusion that I did: it doesn’t make sense. Get real: lots of companies have tried before and failed. I believe that If VMware were to use Zimbra to compete against Exchange in the traditional corporate messaging market, where Exchange mopped up very worthy competitors like Lotus Notes, VMware would end up a failure like so many others before them. As Chris Wolf pointed out on Twitter, the integration between Exchange and the Microsoft applications is just too great to take them head-on.

However, what if you consider that the market is shifting? There is a greater move toward private clouds running scalable, web-based applications. There is an inclination toward workloads that can run outside of an organization’s data center. There is a shift toward virtual desktops. There is desire and interest in embracing the idea of cloud computing—however you choose to define that—across organizations of many different shapes and sizes. In the light of these market factors, now the Zimbra acquisition starts to make more sense. Yes, VMware will compete against Microsoft with Zimbra, but not using today’s architectures and today’s paradigms. As Chris put it in a Twitter post yesterday (emphasis mine):

@TonyWilburn @scott_lowe – IMO vmware has to redefine the traditional app stack for long term survival; email has to be part of it

In the light of this line of thinking, the acquisition begins to make a bit more sense (again, assuming that it’s actually going to happen). Although VMware has different leadership, and it’s a different market, I do feel that VMware would not be successful taking on Microsoft Exchange without redefining how e-mail platforms—as a key part of the overall application stack—can be provisioned, deployed, and managed in conjunction with VMware’s broader private cloud/public cloud strategy.

So what do you think?

Tags: , , , , ,

Vote Early (But Don’t Vote Often)

Monday, January 4, 2010 in General, Networking, Storage, Virtualization by slowe | 1 comment

The time has come again for readers to vote for Eric Siebert’s top 20 bloggers list. I’ve been honored to be placed at #2 for quite some time now, behind Duncan Epping and ahead of Chad Sakac. In addition, some great new bloggers have arrived on the scene this past year, so the competition is going to be fierce. I hope to retain or improve my position, but I’m also realistic—there are some really great bloggers out there.

In addition, my blogging frequency was down this past year because of the two books I wrote this year, Mastering VMware vSphere 4 and VMware vSphere 4 Administration Instant Reference (with Jase McCarty and Matthew Johnson).

Still, the site has a sizable following. I’m averaging around 7,700 RSS subscribers, and the site is getting around 100,000 visits a month (depending upon which statistics mechanism you use, sometimes higher and sometimes lower). Anecdotally, I’ve had a few other bloggers tell me that their traffic spikes whenever I link to one of their posts. I guess I must be doing something right!

Luckily, I have gotten a few good posts out this year:

Creating a Bootable ESXi USB Stick on Mac OS X
vSphere Virtual Machine Upgrade Process
VMware vSphere vDS, VMkernel Ports, and Jumbo Frames
New User’s Guide to Configuring VMware ESX Networking via CLI
Introduction to Nehalem Memory (by guest author Aaron Delp)
Using VMware ESX Virtual Switch Tagging with HP Virtual Connect
Another Reason Not to Use PVSCSI or VMXNET3

And those are just the virtualization-related posts…I’ve also expanded into more SAN-related posts (a couple of Cisco MDS posts here and here), an introductory post on SR-IOV here, and some FCoE posts (here and here, for example).

So, if you could find it in your hearts to vote for me, I’d certainly appreciate it. Go vote now!

Tags: , , ,

Staying in Touch with Me

Monday, January 4, 2010 in General by slowe | 3 comments

With the job change, some of my contact information is also changing. Naturally, my ePlus.com e-mail address will go away, but also going away (effective immediately) are my three ePlus-related instant messaging accounts:

MSN Messenger (slowe@eplus.com)
Yahoo Messenger (eplusslowe)
AOL Instant Messenger (eplusslowe)

If you are using my ePlus e-mail address or any of these instant messaging accounts to stay in touch with me, you’ll need to update your contact information. I don’t have my official EMC e-mail address yet, but my personal e-mail address is available on the About page of this site. I also have Google Talk, AIM, and MSN Messenger accounts that use my personal e-mail address as the user ID; you can use these as a replacement for the ePlus-related instant messaging accounts. I won’t be creating a new Yahoo Messenger account.

If you have my Google Voice number, you can continue using that number to reach me via telephone or SMS. Otherwise, all my old phone numbers will go away. And no, I’m not going to post my Google Voice number out here for anyone to use. Those of you that need it should already have it.

My Twitter ID (scott_lowe) and my blog URL will remain unchanged. I will continue to actively use both to share useful technical information.

Tags: