blog.scottlowe.org

Here’s the latest installation of Virtualization Short Takes, my occasionally-weekly view on various virtualization news, reviews, and other happenings. Hopefully I can share something interesting with you!

• Via VMblog.com, I saw that Transitive Corporation is supporting the use of QuickTransit within Hyper-V virtual machines. This is interesting because it extends the ability of Hyper-V to help customers consolidate applications. QuickTransit, in case you aren’t aware, allows applications written for Solaris/SPARC environments to run in Linux/x86 environments. It was also the technology behind Apple’s Rosetta, which allowed Mac users to run PowerPC apps on Intel Macs. Does anyone know if QuickTransit is supported within VMware VMs, or is this specific to Hyper-V?
• This one was quite interesting to me. Question #2 is particularly applicable: why is a reboot required, anyway? (Yes, yes, I know—there is a workaround that does not require a reboot. It’s the principle of the matter.)
• Via various sources on the Internet, I learned about the release of ESX Manager. This looks like quite an interesting tool, although I have not yet had the opportunity to install or try it yet. Anyone out there tried this and have some feedback for us?
• Every now and then, something comes up about Citrix XenServer and Xen and it makes me wonder about the relationship between Citrix and the open source Xen community. The latest thing is what appears to be an offhand comment by Simon Crosby of Citrix where he says, “Because we own the hypervisor, we can do much more integration and development around it” (read it in context here). What does that mean? What does “ownership” of the Xen hypervisor mean? And if the Xen hypervisor is licensed under an open source license (GNU GPL v2, according to this page), how can Citrix make proprietary extensions to the hypervisor without being forced to release those extensions back to the community? I guess I just don’t understand the relationship there and how it works. This is where the murky waters of a commercial entity “owning” an open source project come into play, in my mind.
• I ran across this very useful tip for creating a vSwitch with a specific number of ports. It looks like Dwight Hubbard, the maintainer of the site, also has some other interesting posts. Might be worth adding his feed to your RSS reader.
• Nick Triantos discusses NetApp’s Site Recovery Adapter (SRA) and its role with VMware Site Recovery Manager (SRM). Anyone have any links to similar discussions of the SRAs for other storage vendors?
• John Howard provides a great breakdown of how Hyper-V generates dynamic MAC addresses and how Hyper-V attempts to protect against MAC collisions in some circumstances.
• The VI3 Security Hardening Guide has been updated, which is good because some people felt it just didn’t go far enough.
• VMware re-iterated their stance on being storage protocol agnostic, and in the article included a very useful table that summarizes the various products and technologies and which are supported with which storage protocols. While the rest of the post is helpful, that summary of supported features is probably the most helpful.
• Interesting in trying out Hyper-V, but don’t have shared storage? Take a look at this blog post. I think you’ll find it helpful.

I’m always on the lookout for other interesting or useful virtualization news, tips, and tricks, so feel free to share with me and other readers in the comments.

I just wanted to provide a quick update on some articles I have in the works to be (hopefully) published soon.

• I’m working on an article discussing when to use various NIC teaming configurations with VMware ESX. There are some significant repercussions here for a variety of network configurations, but especially so for configurations involving IP-based storage (iSCSI or NFS).
• I’m finally wrapping up an article on the Xsigo I/O Director. I’ve been working a Xsigo VP780 in the lab for quite some time, and this article will provide a brief overview along with some tips and tricks.
• I received word from HP that I should be getting a ProCurve switch in my lab soon, so that means I can provide a ProCurve-oriented version of this NIC teaming and VLAN trunking article.
• I have some notes on using NetApp Open Systems SnapVault (OSSV) in conjunction with VMware ESX that I plan to post here as well.

New versions of the Linux and Solaris AD integration articles are on the way as well, starting with an update of the Solaris instructions to accommodate Solaris 10 Update 5 and Windows Server 2008.

If there’s anything else you’re interested in seeing, let me know in the comments. Thanks for reading!

UPDATE: The NIC utilization article is available here.

My Virtualization Short Take series seems to be reasonably popular, so I thought I’d expand into another area that interests me: storage. Here is Storage Short Take #1! If this proves helpful or useful to readers, I’ll continue the series on an irregular basis.

• If you’re new to SnapMirror, especially synchronous SnapMirror, this synchronous SnapMirror configuration guide may prove very helpful to you.
• Good friend Nick Triantos discusses NetApp’s Storage Recovery Adapter (SRA) for VMware Site Recovery Manager (SRM). While the discussion is specific to NetApp, it’s a good example of how the storage vendors are responsible for implementing vendor-specific functionality in the SRA. The other storage vendors supporting SRM are responsible for the same things for their storage arrays and storage array functionality.
• Isn’t this the truth—everyone and their brother has “storage virtualization” functionality built into their products these days. Frankly, I’m tired of hearing about it.
• If you’re running VMs on NFS on NetApp storage, you’ll want to note this knowledge base article (NOW login required). It notes that a SCSI disk timeout increase may need to be set in order to accommodate cluster failover times.
• I recently came across this white paper from Emulex and Cisco regarding the use of N_Port ID Virtualization (NPIV) in a VMware environment. Personally, I found it to be a bit light on the technical details and a bit heavy on the marketing side, but otherwise useful.
• This tool from NetApp (NOW login required) can help with approximating storage I/O. It’s not perfect, but it might help provide some rough estimates. I’m sure other vendors have similar tools; readers are encouraged to share links to those tools in the comments.

Well, that’s it for now. I’d love to hear feedback (good or bad) from readers as to whether this is even remotely useful or interesting.

A couple of days ago my good friend Nick Triantos—formerly of Storage Foo fame, now writing at the Storage Nuts & Bolts Blog—published a short piece of a new version of Open Systems SnapVault (OSSV) that offers new functionality when used with VMware Infrastructure 3 (VI3).

For those that aren’t familiar with OSSV, it’s basically a way to bring NetApp-style snapshots to non-NetApp storage. For example, it’s pretty common to use OSSV to take snapshots of a server that is not attached to a NetApp SAN and store those snapshots on a NetApp SAN. Think of a remote office/branch office kind of scenario, for example, where the branch server can be easily backed up using incremental snapshots to a NetApp storage system in the main location. It’s pretty handy technology, to be honest.

After reading Nick’s piece on OSSV, one question popped in my mind: is he talking about running OSSV on the guest, or running OSSV on the ESX host? So I contacted Nick, obtained clarification, and wanted to post that clarification here. (BTW, thanks, Nick, for taking the time to answer my questions and allowing me to discuss this here.)

Nick’s article primarily focuses on the use of OSSV at the ESX level, running within the ESX Console OS (or Service Console). In this respect, OSSV will be taking snapshots of the VMDK files via the Service Console and replicating them over to a NetApp storage system. As with running OSSV in other environments, the snapshots are block-level incremental snapshots and thus only the changed blocks will be replicated across the wire when a snapshot is taken and shipped off to the destination storage system.

What’s apparently new here is that NetApp has optimized the OSSV code so that the initial baseline will only capture the utilized portions of a VMDK file. Keep in mind that ESX, by default, uses thick-provisioned disks, so that a 30GB virtual disk takes up 30GB on the physical storage. OSSV 2.6, the new version, understands that if only 10GB of that 30GB VDMK is actually being utilized, it will only replicate 10GB of data. That is really important in reducing the overhead required for the initial baseline transfer. Thereafter, OSSV operates as usual by capturing and replicating only changed blocks.

This is a nice addition to OSSV, and it greatly increases the usefulness of OSSV in VI3 environments. But there is a drawback…has anyone figured it out yet?

If you guessed that taking block-level incremental snapshots of the VMDK files via the Service Console means that we lose file-level granularity within the guest file system, you would be correct! What does this mean? Basically, it means that if you have to restore a VMDK from an OSSV snapshot, you have to restore the entire VMDK. You can’t restore individual files within a guest from an OSSV snapshot taken while OSSV is running at the ESX layer.

Before you start knocking OSSV as worthless, however, consider that VCB suffers the same limitation when creating image-level backups. Also keep in mind that file-level backups are only possible with VCB when the guest is running Microsoft Windows, so you’re forced to use image-level backups with any other operating system. Also keep in mind that file-level backups with VCB are slower than image-level backups. With these considerations in mind, it becomes clearer that the limitation is not inherent to OSSV per se, but rather a limitation of technologies operating at the ESX layer.

Of course, there are a number of workarounds to this; one way is to attach the restored VMDK to a different VM, then pull the individual files out that are needed. If you do need file-level granularity from within the guest OS (such as the ability to quickly and easily restore a specific file within a guest), then you can always run OSSV inside the guest and replicate block-level incremental snapshots from the guest over to a target NetApp storage system. Just be sure to keep these distinct procedures in mind as you plan backups of VMs using OSSV.

As always, I encourage you to ask questions, make comments, or add your thoughts below.

UPDATE: Keep in mind that OSSV is VMotion aware, meaning that block-level incrementals are preserved even after a VMotion operation. This is true even within DRS/HA clusters; you just have to install the OSSV agent on all ESX servers within the cluster. The real question: what effect will Storage VMotion have?

The recent couple of articles I wrote about using NetApp deduplication—in particular, the article on using NetApp deduplication with block storage—have raised some questions that are probably worth addressing. Although NetApp deduplication works just fine with block-based storage, there are some considerations with regards to how the LUNs should be provisioned when deduplication will come into play.

Fortunately for me, someone at NetApp decided that it would be a good idea to document the five basic configurations of using NetApp deduplication with block storage. As seen in this comment to an earlier article, Larry Freeman points out this document on the NetApp Communities site (has anyone else noticed the similarity between VMware Communities and NetApp Communities?) that outlines the 5 basic configurations and where the freed blocks go in each configuration. Excellent—that saves me some work!

The most common configurations I’ve seen are configurations D (LUN not space reserved, space guarantee set to Volume) and E (LUN not space reserved, space guarantee set to None). Customers like to see the LUNs “shrink” after deduplication, and this is the only way to make that happen.

The only things we need now are for NetApp to a) remove the volume size limitations and b) get us deduplication at the aggregate level. Then we’d really be set!

What I had hoped to be able to publish today would be an article describing how to configure and use ESX’s software iSCSI initiator as a failover path for Fibre Channel, so that if the Fibre Channel fabric completely failed VM traffic would automatically failover to software iSCSI. I thought that this would be a great, low-cost way to add another layer of redundancy to your VMware ESX environment.

Unfortunately, I can’t make it work. Here’s the setup I’ve been using for testing:

• A 200GB LUN visible to ESX over both Fibre Channel (FC) and software iSCSI
• A VM, stored on this LUN, running Windows Server 2003 R2

Initial tests led me to believe that it would indeed work. I verified that both the FC path as well as the iSCSI path were listed as separate paths for the same LUN. Without placing any load on the VM, I pulled the FC connection from the back of the server. The VM stayed up, and I was able to browse the local hard drive inside the VM. Network connectivity remained active. And the “Manage Paths” dialog box even showed the FC connection as “Dead” and the iSCSI connection as On/Active. Given that information, it seemed like all was good.

Determined to verify that it was working as I expected, I trotted out a copy of IOmeter and tried to repeat the tests. This time around, though, the tests did not go quite so well. IOmeter showed that disk throughput stopped, and the VI Client locked up. I repeated this set of tests a couple of times, and each time—while IOmeter was running—I ran into issues.

Based on these results, I’m inclined to say that one of two things is true. Either:

1. I did something very, very wrong; or
2. ESX isn’t quite right to support automatic failover between FC and software iSCSI.

Has anyone else tried this, or am I the only one? If you have tried it, did it work? If so, what steps did you have to take—if any—to make it work properly?

Building on my earlier article on setting up NetApp deduplication, I wanted to follow up with some information on using NetApp deduplication with block storage (LUNs presented via Fibre Channel or iSCSI).

For the most part, using NetApp deduplication with block storage is a lot like I described earlier:

• You (obviously) still need the NearStore and deduplication (A-SIS) licenses installed on the controller(s).
• You will still turn deduplication on using the “sis on” command for the FlexVol containing the LUNs.
• Limitations on the size of the FlexVol still apply.
• You use the “sis status” command to check on the status of deduplication, and the “sis config” command to see the deduplication schedule.

OK, so what’s different? Well, it has to do with how LUNs are provisioned on a NetApp storage system. I’ve blogged before about managing LUN space requirements on a NetApp, and about using LUN clones vs. FlexClones. That second article, in particular, really goes into detail on how LUNs are implemented on top of NetApp’s file system, WAFL. Since LUNs are represented by WAFL as a single file, they are also normally “space reserved,” meaning that the maximum size of the LUN is allocated at the time of creation. If you create a 50GB LUN, then Data ONTAP creates a 50GB file right away. (For readers out there who are well-versed in NetApp storage, I know that’s a bit of a simplification, but bear with me.)

What does this have to do with deduplication? Great question. If the LUN is space reserved—meaning that the maximum size of the LUN is allocated up front and remains allocated to the LUN—then the file that represents the LUN won’t ever decrease in size to reflect deduplication savings, and deduplication therefore does you absolutely no good whatsoever. This is not to say that deduplication doesn’t work, just that it won’t help you at all.

Fortunately, there’s an easy fix for this. When creating the LUN, simply uncheck the box marked “Space Reserved” and allow Data ONTAP to allocate space to the LUN out of the containing FlexVol on an as-needed basis. Because the file that represents the LUN can grow in size, it can also shrink in size, and deduplication will cause the file that represents the LUN to decrease in size. This then allows you to provision additional LUNs from the same FlexVol to take advantage of the space savings resulting from deduplication.

I know that seems a bit confusing; I’ll probably post another article with some more in-depth discussions of the details. (Either that, or I’ll encourage my NetApp readers to chime in below in the comments.)

So, in summary, when using NetApp deduplication with block storage:

• you’ll setup and configure deduplication on the FlexVol containing your LUN(s) just like described in my earlier article;
• you’ll uncheck the “Space Reserved” checkbox when creating the LUNs to be deduplicated;
• you won’t see the space savings from the host’s perspective and therefore can’t store more data in that LUN than the size of the LUN; but
• you will be able to provision additional LUNs in that same FlexVol that can be presented back to host for additional storage.

I hope this helps clarify some of the questions or issues surrounding the use of NetApp deduplication with block storage. Feel free to add information, experiences with deduplication and block storage, or ask additional questions in the comments below.

UPDATE: There are some additional considerations about how to provision LUNs along with NetApp deduplication that warrant a more in-depth discussion. Look for a follow-up post within the next few days.

With the release of VMware Infrastructure version 3.5, VMware added support for jumbo frames. Although the documentation states that jumbo frames “are not supported for NAS and iSCSI traffic”, jumbo frames for NFS and iSCSI does actually work. Here’s some information on getting it working.

How I Tested

Keep in mind that this is not an “officially supported” configuration (see the section on the “Official” Support Statement below), so use at your own risk. I will not be held responsible if you blow up your production environment trying to make jumbo frames work.

Here’s how I tested the use of jumbo frames for software iSCSI and NFS datastores:

• For the physical switch infrastructure, I used a Cisco Catalyst 3560G running Cisco IOS version 12.2(25)SEB4.
• For the physical server hardware, I used a group of HP ProLiant DL385 G2 servers with dual-core AMD Opteron processors and a quad-port PCIe Intel Gigabit Ethernet NIC.
• For the storage system, I used a NetApp FAS940 running Data ONTAP 7.2.4.

The exact commands and/or procedures may be different for you depending upon the hardware and/or software versions that you’re running in your environment. Keep that in mind.

Configuring the Physical Switch

Fortunately for me, the Cisco Catalyst 3560G does indeed support jumbo frames. (Naturally, you’ll want to ensure that your switch supports jumbo frames.) Jumbo frames are not, however, enabled by default; they must be enabled using the following command in global configuration mode:

system mtu jumbo 9000

Note that 9000 bytes seems to be the generally accepted size for jumbo frames, so that’s what I used.

After running this command, you must reboot the switch. The change doesn’t take effect until a reload. Fortunately, IOS reminds you of this after you enter the command. Once the switch has rebooted, you can verify the MTU setting with this command:

show system mtu

This should report that the system jumbo MTU size is 9000 bytes, confirming that the switch is ready for jumbo frames. Now we’re prepared to configure the storage system.

Configuring the Storage System

Using FilerView, increasing the MTU on the appropriate network interfaces to 9000 bytes is as simple as going to Network > Manage Interfaces and then clicking the Modify link for the interface to be changed. Set the “MTU size” to 9000 (from the default of 1500), click Apply, and you’re ready to roll.

You can verify the settings in FilerView using Network > Manage Interfaces > Show All Interface Details, or by using the “ifconfig -a” command from a Data ONTAP command prompt.

Configuring ESX Server

There is no GUI in VirtualCenter for configuring jumbo frames; all of the configuration must be done from a command line on the ESX server itself. There are two basic steps:

1. Configure the MTU on the vSwitch.
2. Create a VMkernel interface with the correct MTU.

First, we need to set the MTU for the vSwitch. This is pretty easily accomplished using esxcfg-vswitch:

esxcfg-vswitch -m 9000 vSwitch1

A quick run of “esxcfg-vswitch -l” (that’s a lowercase L) will show the vSwitch’s MTU is now 9000; in addition, “esxcfg-nics -l” (again, a lowercase L) will show the MTU for the NICs linked to that vSwitch are now set to 9000 as well.

Second, we need to create a VMkernel interface. This step is a bit more complicated, because we need to have a port group in place already, and that port group needs to be on the vSwitch whose MTU we set previously:

esxcfg-vmknic -a -i 172.16.1.1 -n 255.255.0.0 -m 9000 IPStorage

This creates a port group called IPStorage on vSwitch1—the vSwitch whose MTU was previously set to 9000—and then creates a VMkernel port with an MTU of 9000 on that port group. Be sure to use an IP address that is appropriate for your network when creating the VMkernel interface.

To test that everything is working so far, use the vmkping command:

vmkping -s 9000 172.16.1.200

Clearly, you’ll want to substitute the IP address of your storage system in that command.

That’s it! From here you should be able to easily add an NFS datastore or connect to an iSCSI LUN using jumbo frames from the ESX server.

“Official” Support Statement

Officially, jumbo frames are only supported by VMware for use by virtual machines. Technically, VMware does not support the use of jumbo frames for the software iSCSI initiator or for use with NFS datastores. At least, that’s my understanding.

So, feel free to tinker around with jumbo frames for IP-based storage, and when VMware adds official support for it in the future—I can’t imagine why they wouldn’t—then you’ll be able to hit the ground running with the configuration steps necessary to make it work.

VMware’s done a great job of rolling together the VMware news and views with their VMware blog aggregator, Planet V12n. Does anyone know of something equivalent for NetApp? Where is “Planet NetApp”?

A short while ago, I discussed that VMDKs on NFS may start out thin provisioned, but will lose that thin provisioned status over time. Operations like cloning and Storage VMotion will cause these thin provisioned disks to become thick (fully provisioned) disks, and you lose one of the benefits of running VMware on NFS.

Fortunately, if you’re running a NetApp storage system as your NFS server, you can preserve the thin provisioned status of these VMDKs by leveraging NetApp’s single file SnapRestore functionality. This article describes how that works.

There’s a couple of caveats here:

1. This technique only helps with making new VMs from an existing VM. SnapRestore won’t help preserve thin provisioned status after a Storage VMotion operation.
2. This isn’t integrated with VirtualCenter, so you won’t be able to take advantage of VirtualCenter’s integration with Sysprep and such.

As a result of #2 above, then, you’ll need to first prepare your source VM by running Sysprep inside the VM (assuming it is a Windows-based VM) and then allowing Sysprep to shut down the VM. Once that’s accomplished, then you can proceed.

The first step is to take a snapshot of the volume containing the already prepared VM:

snap create <vol-name> <snapshot-name>

Next, create a new VM in VirtualCenter, but do not create a virtual disk for the VM. This will create the VM configuration and associated files and the directory on the NFS datastore.

Third, run a SnapRestore operation to restore both the .vmdk file and the -flat.vmdk files. You have to restore both in order for this to work; keep in mind that the .vmdk file is just a header file and the -flat.vmdk is the actual disk file. The commands would look something like this:

snap restore -t file -s <snapshot-name> -r <new filename and path> <original filename and path>

As an example, let’s say you had a VM named template01 and you wanted to clone the disks for template01 to a new VM called newvm01, and these are stored on a volume called nfsvol. After you’ve run Sysprep on template01, taken the Snapshot and called it base_snapshot, and created newvm01 without a virtual disk, you’d run this command:

snap restore -t file -s base_snapshot -r /vol/nfsvol/newvm01/newvm01.vmdk /vol/nfsvol/template01/template01.vmdk

That would restore the .vmdk (header) file; then you’d restore the actual virtual disk file:

snap restore -t file -s base_snapshot -r /vol/nfsvol/newvm01/newvm01-flat.vmdk /vol/nfsvol/template01/template01-flat.vmdk

Once this process is complete—and it may take some time depending upon the size of the files being restored—you should see both the .vmdk and the -flat.vmdk files listed in the Datastore Browser.

“But wait a minute, Scott,” you say. “The -flat.vmdk no longer looks thin provisioned. You lied! This process doesn’t work.”

Indeed, it will look like it is no longer thin provisioned. Trust me; there’s one more step and then all will make sense. If you log into the ESX Server and open the .vmdk file in vi, you’ll see that it references the old file name of the -flat.vmdk. Edit that to reflect the new, restored file name, save your changes, and go back to the Datastore Broswer again. Refresh the display, and all should be well.

Why does the Datastore Browser work that way? Beats me. You’ll also find that running an “ls -la” on an NFS datastore from the Service Console will show you -flat.vmdk files that appear to be thick provisioned. The only way I’ve found to see if they are thin provisioned is to use the Datastore Browser. It’s a VMware thing, I suppose.

The last and final step is to edit the settings for the VM you created earlier and add the new virtual disk to that VM. Then you can boot up that VM and proceed with whatever customization steps, if any, are needed.

In the near future I plan to test another possible method of preserving the VMDK’s thin provisioned status, a method that is storage agnostic. Look for details of that testing here.