blog.scottlowe.org

This one is a short one, since I’m out of town on business and don’t have a great deal of time. Nevertheless, I wanted to post my thoughts on a couple of items that have passed in front of me over the last few days.

• Andrew Kutz has released the VI Toolkit for .NET. Quoting from the e-mail he sent me, the VI Toolkit for .NET “makes connecting to VI, finding a VM, and getting its properties a snap!”. If you’re a developer that is interested in developing tools and/or scripts for the VI environment, this seems like it would be a tool that would be useful.
• VKernel let me know that they’ve joined the VMware Certified Virtual Appliance program with their Chargeback Virtual Appliance.
• Rich over at VM /ETC has published some information on using the VI Client to upgrade VMware Tools. The particularly helpful portion of this article is the information on using the CLI from the VirtualCenter server to upgrade VMware Tools.
• Duncan shares some information from one of his readers about a bug in the VMware Tools for Solaris 10. This particular issue can prevent VCB from working properly. Fortunately, Duncan also shares a workaround.
• Via Alessandro, it looks like Microsoft is upping the heat on VMware with the release of a beta of SCVMM 2008. One of the most intriguing features to be included in the next version is management from SCVMM of VMware ESX Server. When you’re a company like Microsoft, you can afford to wage a battle on two fronts—in this case, on both the hypervisor front and the hypervisor management front.
• Again via Duncan, it looks like there’s an error with ESX 3.5 Update 1 with the Pegasus installation. I ran into this error myself when installing Update 1 right after it was released, but then marked it up as a symptom of the whole snafu with the wrong version of the ISO being posted. Looks like it was more than that.
• Rich has some good information on ESX-VirtualCenter compatibility. (Yes, I do read more blogs than just Rich’s and Duncan’s. Honest!)
• OK, this may not be necessarily virtualization related, but a pretty good storage-related blog, which does address some VMware-specific issues, has sprouted up here.
• SearchVMware.com has published an interesting article by David Davis about using ESX to help with an Exchange migration or Exchange upgrade.
• If you have any questions about networking performance on ESX, just check out this document, last revised close to the end of February of this year.

Well, that’s it for now. Thanks for reading!

SearchServerVirtualization.com has published an article I wrote on the emerging field of I/O virtualization and the potential benefits to a server virtualization environment:

I/O virtualization is a new form of virtualization that is justifiably gaining attention in the data center. It’s one of several virtualization offshoots — such as service virtualization or facilities virtualization — spawned by the meteoric rise of server virtualization. But what is I/O virtualization, exactly? And what benefits can it bring to the data center?

Read the full article for all the details!

What I had hoped to be able to publish today would be an article describing how to configure and use ESX’s software iSCSI initiator as a failover path for Fibre Channel, so that if the Fibre Channel fabric completely failed VM traffic would automatically failover to software iSCSI. I thought that this would be a great, low-cost way to add another layer of redundancy to your VMware ESX environment.

Unfortunately, I can’t make it work. Here’s the setup I’ve been using for testing:

• A 200GB LUN visible to ESX over both Fibre Channel (FC) and software iSCSI
• A VM, stored on this LUN, running Windows Server 2003 R2

Initial tests led me to believe that it would indeed work. I verified that both the FC path as well as the iSCSI path were listed as separate paths for the same LUN. Without placing any load on the VM, I pulled the FC connection from the back of the server. The VM stayed up, and I was able to browse the local hard drive inside the VM. Network connectivity remained active. And the “Manage Paths” dialog box even showed the FC connection as “Dead” and the iSCSI connection as On/Active. Given that information, it seemed like all was good.

Determined to verify that it was working as I expected, I trotted out a copy of IOmeter and tried to repeat the tests. This time around, though, the tests did not go quite so well. IOmeter showed that disk throughput stopped, and the VI Client locked up. I repeated this set of tests a couple of times, and each time—while IOmeter was running—I ran into issues.

Based on these results, I’m inclined to say that one of two things is true. Either:

1. I did something very, very wrong; or
2. ESX isn’t quite right to support automatic failover between FC and software iSCSI.

Has anyone else tried this, or am I the only one? If you have tried it, did it work? If so, what steps did you have to take—if any—to make it work properly?

In his three recent articles about Quick Migration and live migration, Jeff Woolsey spends a lot of effort differentiating Quick Migration, VMotion, and VMware HA. Personally, I thought the distinction between these features and the purposes they were intended to serve were pretty clear already, but apparently Microsoft’s earlier claims that Quick Migration and live migration were comparable confused everyone.

The three articles from Jeff can be found here:

Hyper-V Quick Migration and VMware Live Migration, Part 1
Hyper-V Quick Migration and VMware Live Migration, Part 2
Hyper-V Quick Migration and VMware Live Migration, Part 3

In the first article, Jeff discusses the importance of high availability (HA) in virtualization scenarios. He’s absolutely right on target with his statements: HA is critical in virtualization implementations. I couldn’t agree more. VMware recognizes this fact and includes VMware HA, and Microsoft recognizes this fact and provides integration between Hyper-V and Windows Server 2008 Failover Clustering. So far, so good.

In part two, Jeff goes on to state that VMotion doesn’t work for unplanned downtime. Again, he’s absolutely correct: VMotion doesn’t work for unplanned downtime. Then again, apart from the comments that Jeff claims to have received from VMware supporters stating VMotion was “far superior for unplanned host downtime and that it was a much better HA solution”, I don’t think anyone has ever claimed that VMotion was an HA solution. I know I certainly haven’t. I can’t recall VMware ever making that statement. After all, if VMotion were an HA solution, why would VMware have VMware HA? What point would there be in two different HA solutions?

Further in that same article, Jeff compares VMware HA with Windows Server 2008 Failover Clustering, aka Quick Migration, and states that they are comparable technologies. Once again, I agree; VMware HA and Quick Migration are comparable technologies. Both will restart virtual machines on another host automatically in the event of a host failure. OK, Jeff and I still agree thus far.

Part three of Jeff’s series wraps things up by attempting to downplay the importance of VMotion. In his words, “Even customers with Live Migration still wait until off hours to service the hardware.” Unfortunately, this is where Jeff and I have to disagree. I don’t know how many customers they spoke to, but I know I have customers that have live migration functionality who use it during business hours. Besides, live migration isn’t just about hardware servicing or patching the root partition/parent partition/Service Console, it’s also about enabling dynamic load balancing a la VMware DRS, or enabling power savings in off-hours via DPM. After all, just because you can shut down and/or power off guests to migrate them doesn’t mean you will necessarily want to in every instance. It may be acceptable for some workloads, but not for all workloads.

I just can’t help but feeling that if Microsoft hadn’t made the comparisons between VMotion and Quick Migration themselves earlier in Hyper-V’s development, this sort of “unequal comparison” that Jeff is trying so hard to clear up wouldn’t have happened.

VMware ESX Server in the Enterprise: Planning and Securing Virtualization Servers, by Edward Haletky, is a book I’ve been working on reviewing for quite a while now. It’s a fairly hefty tome, weighing in at just over 550 pages, and is chock full of technical details on both ESX Server 2.5.x and ESX 3.0.x. Throughout the book, the author faithfully covers information for both versions—where applicable, of course—and highlights differences and similarities.

Although I personally found the constant “back and forth” between ESX Server 2.5.x and ESX 3.0.x to be distracting, I can easily see where readers fresh out of an ESX Server 2.5.x migration—or still managing some servers running ESX Server 2.5.x—would find that aspect of the book useful. VMware administrators familiar with the older version but perhaps just now making the move to VI3 would also find the book useful as a “transition” manual.

I did run into a few technical inaccuracies, but these are minor in scope and do not materially affect the content. For example, on page 180 in the section titled “iSCSI/NFS Best Practices,” the author makes the following statement:

The iSCSI VMkernel device must be part of at least one service console vSwitch, which implies that the service console must be on the same network as the iSCSI servers. This is required whether using CHAP authentication, or not, to pass credentials to the iSCSI server.

Technically, the iSCSI VMkernel device can be on any vSwitch; the requirement is that there is Service Console connectivity to the iSCSI target. This connectivity could be direct (on the same subnet) or routed. While following the author’s guidance and placing a Service Console port group on the same vSwitch and same IP network as the iSCSI VMkernel device will most certainly work, it’s not required. It’s a very minor inaccuracy, as I said earlier, and does not substantially or materially change the validity of the material.

Summary

Overall, I found the book to be good reference material. Haletky covers a broad range of topics, from installation to storage and networking to disaster recovery. Anyone needing reference material for such a wide range of topics could do far worse than choosing this book.

VMware has released an update to VMware Fusion, their Mac OS X-based virtualization product. Version 1.1.2 includes a number of fixes and new features. More information is available here.

If you’re a Fusion user, go grab the free update.

I installed the update first thing this morning. I wasn’t suffering from any of the problems that were fixed in this version, so I can’t attest to that, but I can say that resuming and suspending VMs seems quicker. Perhaps it’s just my imagination…

Building on my earlier article on setting up NetApp deduplication, I wanted to follow up with some information on using NetApp deduplication with block storage (LUNs presented via Fibre Channel or iSCSI).

For the most part, using NetApp deduplication with block storage is a lot like I described earlier:

• You (obviously) still need the NearStore and deduplication (A-SIS) licenses installed on the controller(s).
• You will still turn deduplication on using the “sis on” command for the FlexVol containing the LUNs.
• Limitations on the size of the FlexVol still apply.
• You use the “sis status” command to check on the status of deduplication, and the “sis config” command to see the deduplication schedule.

OK, so what’s different? Well, it has to do with how LUNs are provisioned on a NetApp storage system. I’ve blogged before about managing LUN space requirements on a NetApp, and about using LUN clones vs. FlexClones. That second article, in particular, really goes into detail on how LUNs are implemented on top of NetApp’s file system, WAFL. Since LUNs are represented by WAFL as a single file, they are also normally “space reserved,” meaning that the maximum size of the LUN is allocated at the time of creation. If you create a 50GB LUN, then Data ONTAP creates a 50GB file right away. (For readers out there who are well-versed in NetApp storage, I know that’s a bit of a simplification, but bear with me.)

What does this have to do with deduplication? Great question. If the LUN is space reserved—meaning that the maximum size of the LUN is allocated up front and remains allocated to the LUN—then the file that represents the LUN won’t ever decrease in size to reflect deduplication savings, and deduplication therefore does you absolutely no good whatsoever. This is not to say that deduplication doesn’t work, just that it won’t help you at all.

Fortunately, there’s an easy fix for this. When creating the LUN, simply uncheck the box marked “Space Reserved” and allow Data ONTAP to allocate space to the LUN out of the containing FlexVol on an as-needed basis. Because the file that represents the LUN can grow in size, it can also shrink in size, and deduplication will cause the file that represents the LUN to decrease in size. This then allows you to provision additional LUNs from the same FlexVol to take advantage of the space savings resulting from deduplication.

I know that seems a bit confusing; I’ll probably post another article with some more in-depth discussions of the details. (Either that, or I’ll encourage my NetApp readers to chime in below in the comments.)

So, in summary, when using NetApp deduplication with block storage:

• you’ll setup and configure deduplication on the FlexVol containing your LUN(s) just like described in my earlier article;
• you’ll uncheck the “Space Reserved” checkbox when creating the LUNs to be deduplicated;
• you won’t see the space savings from the host’s perspective and therefore can’t store more data in that LUN than the size of the LUN; but
• you will be able to provision additional LUNs in that same FlexVol that can be presented back to host for additional storage.

I hope this helps clarify some of the questions or issues surrounding the use of NetApp deduplication with block storage. Feel free to add information, experiences with deduplication and block storage, or ask additional questions in the comments below.

UPDATE: There are some additional considerations about how to provision LUNs along with NetApp deduplication that warrant a more in-depth discussion. Look for a follow-up post within the next few days.

In the last few days, I upgraded my work laptop to Office 2008, the latest version of Microsoft’s office suite for Mac OS X. I was looking forward to the Universal binaries (no more sluggish performance due to Rosetta) and Office 2007 file format compatibility. I did get those, but what I also got were some really ugly icons.

So, I did what any enterprising Mac user would do. I replaced them with a new set.

Here’s what the new icons look like:

I found the replacement icons here; look for the set titled “HLD V2 by Henri Liriani”. The icons are well designed and come in a variety of sizes. Upon unpacking the downloaded file, I simply took the .ICNS files in the package and replaced the original files in the application bundles.

For example, right-clicking on Microsoft Entourage and selecting “Show Package Contents,” opening the Contents folder, then opening the Resources folder will show where the new icon needs to be copied. The original icon is in a file named Entourage.icns. Rename this file (just in case you need the old one) and replace it with the downloaded version, making sure that you name the downloaded version with the same original name. Poof—Entourage now has a new icon.

For the other applications, the files to replace are:

Microsoft Word/Contents/Resources/MSWD.icns
Microsoft Excel/Contents/Resources/XCEL.icns
Microsoft PowerPoint/Contents/Resources/PPT3.icns

Replace each of these files, keeping the filename intact, with the new versions.

Once that’s done, you too can enjoy much more pleasant Office 2008 icons!

It appears that some of my articles from a couple of years ago are being republished on another site without any attribution or link back to the original. That’s one of the bad things about the Internet as it stands today…it’s far too easy to “copy and paste” another person’s material onto your own site.

If you’re interested in republishing my content, just drop me a line. I’m sure we can work something out. Just don’t shamelessly steal my writing.

With the release of VMware Infrastructure version 3.5, VMware added support for jumbo frames. Although the documentation states that jumbo frames “are not supported for NAS and iSCSI traffic”, jumbo frames for NFS and iSCSI does actually work. Here’s some information on getting it working.

How I Tested

Keep in mind that this is not an “officially supported” configuration (see the section on the “Official” Support Statement below), so use at your own risk. I will not be held responsible if you blow up your production environment trying to make jumbo frames work.

Here’s how I tested the use of jumbo frames for software iSCSI and NFS datastores:

• For the physical switch infrastructure, I used a Cisco Catalyst 3560G running Cisco IOS version 12.2(25)SEB4.
• For the physical server hardware, I used a group of HP ProLiant DL385 G2 servers with dual-core AMD Opteron processors and a quad-port PCIe Intel Gigabit Ethernet NIC.
• For the storage system, I used a NetApp FAS940 running Data ONTAP 7.2.4.

The exact commands and/or procedures may be different for you depending upon the hardware and/or software versions that you’re running in your environment. Keep that in mind.

Configuring the Physical Switch

Fortunately for me, the Cisco Catalyst 3560G does indeed support jumbo frames. (Naturally, you’ll want to ensure that your switch supports jumbo frames.) Jumbo frames are not, however, enabled by default; they must be enabled using the following command in global configuration mode:

system mtu jumbo 9000

Note that 9000 bytes seems to be the generally accepted size for jumbo frames, so that’s what I used.

After running this command, you must reboot the switch. The change doesn’t take effect until a reload. Fortunately, IOS reminds you of this after you enter the command. Once the switch has rebooted, you can verify the MTU setting with this command:

show system mtu

This should report that the system jumbo MTU size is 9000 bytes, confirming that the switch is ready for jumbo frames. Now we’re prepared to configure the storage system.

Configuring the Storage System

Using FilerView, increasing the MTU on the appropriate network interfaces to 9000 bytes is as simple as going to Network > Manage Interfaces and then clicking the Modify link for the interface to be changed. Set the “MTU size” to 9000 (from the default of 1500), click Apply, and you’re ready to roll.

You can verify the settings in FilerView using Network > Manage Interfaces > Show All Interface Details, or by using the “ifconfig -a” command from a Data ONTAP command prompt.

Configuring ESX Server

There is no GUI in VirtualCenter for configuring jumbo frames; all of the configuration must be done from a command line on the ESX server itself. There are two basic steps:

1. Configure the MTU on the vSwitch.
2. Create a VMkernel interface with the correct MTU.

First, we need to set the MTU for the vSwitch. This is pretty easily accomplished using esxcfg-vswitch:

esxcfg-vswitch -m 9000 vSwitch1

A quick run of “esxcfg-vswitch -l” (that’s a lowercase L) will show the vSwitch’s MTU is now 9000; in addition, “esxcfg-nics -l” (again, a lowercase L) will show the MTU for the NICs linked to that vSwitch are now set to 9000 as well.

Second, we need to create a VMkernel interface. This step is a bit more complicated, because we need to have a port group in place already, and that port group needs to be on the vSwitch whose MTU we set previously:

esxcfg-vmknic -a -i 172.16.1.1 -n 255.255.0.0 -m 9000 IPStorage

This creates a port group called IPStorage on vSwitch1—the vSwitch whose MTU was previously set to 9000—and then creates a VMkernel port with an MTU of 9000 on that port group. Be sure to use an IP address that is appropriate for your network when creating the VMkernel interface.

To test that everything is working so far, use the vmkping command:

vmkping -s 9000 172.16.1.200

Clearly, you’ll want to substitute the IP address of your storage system in that command.

That’s it! From here you should be able to easily add an NFS datastore or connect to an iSCSI LUN using jumbo frames from the ESX server.

“Official” Support Statement

Officially, jumbo frames are only supported by VMware for use by virtual machines. Technically, VMware does not support the use of jumbo frames for the software iSCSI initiator or for use with NFS datastores. At least, that’s my understanding.

So, feel free to tinker around with jumbo frames for IP-based storage, and when VMware adds official support for it in the future—I can’t imagine why they wouldn’t—then you’ll be able to hit the ground running with the configuration steps necessary to make it work.