blog.scottlowe.org

My Virtualization Short Take series seems to be reasonably popular, so I thought I’d expand into another area that interests me: storage. Here is Storage Short Take #1! If this proves helpful or useful to readers, I’ll continue the series on an irregular basis.

• If you’re new to SnapMirror, especially synchronous SnapMirror, this synchronous SnapMirror configuration guide may prove very helpful to you.
• Good friend Nick Triantos discusses NetApp’s Storage Recovery Adapter (SRA) for VMware Site Recovery Manager (SRM). While the discussion is specific to NetApp, it’s a good example of how the storage vendors are responsible for implementing vendor-specific functionality in the SRA. The other storage vendors supporting SRM are responsible for the same things for their storage arrays and storage array functionality.
• Isn’t this the truth—everyone and their brother has “storage virtualization” functionality built into their products these days. Frankly, I’m tired of hearing about it.
• If you’re running VMs on NFS on NetApp storage, you’ll want to note this knowledge base article (NOW login required). It notes that a SCSI disk timeout increase may need to be set in order to accommodate cluster failover times.
• I recently came across this white paper from Emulex and Cisco regarding the use of N_Port ID Virtualization (NPIV) in a VMware environment. Personally, I found it to be a bit light on the technical details and a bit heavy on the marketing side, but otherwise useful.
• This tool from NetApp (NOW login required) can help with approximating storage I/O. It’s not perfect, but it might help provide some rough estimates. I’m sure other vendors have similar tools; readers are encouraged to share links to those tools in the comments.

Well, that’s it for now. I’d love to hear feedback (good or bad) from readers as to whether this is even remotely useful or interesting.

A lot of virtualization-related articles were published while I was in Orlando at Microsoft Tech-Ed 2008. Here are a few that caught my attention over that time period as well as in the last few days.

• Rich Brambley over at VM /ETC is on a roll with a couple of good articles, one on VMFS storage sizing for performance and one on downgrading the VM HAL after P2V. The title for that second article was “How to P2V Multi-processor servers to Uni-processor VMs”, and I was hoping that it was a new trick with VMware Converter or some other P2V tool that would actually take care of the HAL for me. Alas, not this time. Rich’s article is useful information nevertheless, don’t get me wrong. I am curious, though, as to the source of Rich’s storage sizing recommendations. Rich, can you share where you derived those recommendations?
• Ryan Arneson shares some of his experiences in using an X4500 with ZFS as an NFS datastore for ESX. Ryan’s post reminds readers of the some of the requirements for using NFS with ESX, so readers new to that sort of configuration may find it helpful.
• Gabe brings us, courtesy of this VMware Communities thread, some information on getting DRS VMotion information from the VC database. That’s handy. In an earlier article, Gabe also weighed in on storage sizing as well. This seems to be getting quite a bit of attention recently (gee, I can’t imagine why). Readers, I’d love to hear your thoughts on storage sizing approaches, algorithms, etc. Please share them in the comments!
• I found this article discussing NFS vs. CIFS for VMware. At first I was a bit baffled, but then I realized the author must have been talking about VMware’s hosted products like VMware Server not ESX. Anyone else tried this?
• Rick Blythe aka VMwareWolf posted an article about VM customization failing after VC 2.5 upgrade. Fortunately, the fix is easy; just download and install the correct version of the Sysprep tools and put them in the right location on the VC server. The details are all provided at his site, so check that out.
• The VMware Fusion Team announced support for running Mac OS X Leopard Server in Fusion 2.0 back during the WWDC, but it seems that Parallels may have beaten them to the punch with an actually shipping product.
• Schley Andrew Kutz has released a version of the VI Toolkit for .NET that supports mocking. More information on mocking the VI Toolkit is available here. I haven’t seen any sample taunts yet. (You’ll get that in a few minutes.)

In addition, as a follow-up to my Tech-Ed coverage of the Microsoft Assessment and Planning (MAP) toolkit, I found these two articles in my list of “things to blog about”:

Microsoft Assessment and Planning How-To Series: Part 1 (Server Virtualization Candidacy Reporting)
[VIDEO] Microsoft Assessment and Planning How-To Series: Part 1 (Server Virtualization Candidacy Reporting)

Note also that the beta program for MAP 3.1 is already running; this version will add support for Hyper-V. More information here. (I couldn’t remember if I stated anything about this my Tech-Ed session liveblogs.)

That about does it this time around. Thanks for reading!

With the release of VMware Infrastructure version 3.5, VMware added support for jumbo frames. Although the documentation states that jumbo frames “are not supported for NAS and iSCSI traffic”, jumbo frames for NFS and iSCSI does actually work. Here’s some information on getting it working.

How I Tested

Keep in mind that this is not an “officially supported” configuration (see the section on the “Official” Support Statement below), so use at your own risk. I will not be held responsible if you blow up your production environment trying to make jumbo frames work.

Here’s how I tested the use of jumbo frames for software iSCSI and NFS datastores:

• For the physical switch infrastructure, I used a Cisco Catalyst 3560G running Cisco IOS version 12.2(25)SEB4.
• For the physical server hardware, I used a group of HP ProLiant DL385 G2 servers with dual-core AMD Opteron processors and a quad-port PCIe Intel Gigabit Ethernet NIC.
• For the storage system, I used a NetApp FAS940 running Data ONTAP 7.2.4.

The exact commands and/or procedures may be different for you depending upon the hardware and/or software versions that you’re running in your environment. Keep that in mind.

Configuring the Physical Switch

Fortunately for me, the Cisco Catalyst 3560G does indeed support jumbo frames. (Naturally, you’ll want to ensure that your switch supports jumbo frames.) Jumbo frames are not, however, enabled by default; they must be enabled using the following command in global configuration mode:

system mtu jumbo 9000

Note that 9000 bytes seems to be the generally accepted size for jumbo frames, so that’s what I used.

After running this command, you must reboot the switch. The change doesn’t take effect until a reload. Fortunately, IOS reminds you of this after you enter the command. Once the switch has rebooted, you can verify the MTU setting with this command:

show system mtu

This should report that the system jumbo MTU size is 9000 bytes, confirming that the switch is ready for jumbo frames. Now we’re prepared to configure the storage system.

Configuring the Storage System

Using FilerView, increasing the MTU on the appropriate network interfaces to 9000 bytes is as simple as going to Network > Manage Interfaces and then clicking the Modify link for the interface to be changed. Set the “MTU size” to 9000 (from the default of 1500), click Apply, and you’re ready to roll.

You can verify the settings in FilerView using Network > Manage Interfaces > Show All Interface Details, or by using the “ifconfig -a” command from a Data ONTAP command prompt.

Configuring ESX Server

There is no GUI in VirtualCenter for configuring jumbo frames; all of the configuration must be done from a command line on the ESX server itself. There are two basic steps:

1. Configure the MTU on the vSwitch.
2. Create a VMkernel interface with the correct MTU.

First, we need to set the MTU for the vSwitch. This is pretty easily accomplished using esxcfg-vswitch:

esxcfg-vswitch -m 9000 vSwitch1

A quick run of “esxcfg-vswitch -l” (that’s a lowercase L) will show the vSwitch’s MTU is now 9000; in addition, “esxcfg-nics -l” (again, a lowercase L) will show the MTU for the NICs linked to that vSwitch are now set to 9000 as well.

Second, we need to create a VMkernel interface. This step is a bit more complicated, because we need to have a port group in place already, and that port group needs to be on the vSwitch whose MTU we set previously:

esxcfg-vmknic -a -i 172.16.1.1 -n 255.255.0.0 -m 9000 IPStorage

This creates a port group called IPStorage on vSwitch1—the vSwitch whose MTU was previously set to 9000—and then creates a VMkernel port with an MTU of 9000 on that port group. Be sure to use an IP address that is appropriate for your network when creating the VMkernel interface.

To test that everything is working so far, use the vmkping command:

vmkping -s 9000 172.16.1.200

Clearly, you’ll want to substitute the IP address of your storage system in that command.

That’s it! From here you should be able to easily add an NFS datastore or connect to an iSCSI LUN using jumbo frames from the ESX server.

“Official” Support Statement

Officially, jumbo frames are only supported by VMware for use by virtual machines. Technically, VMware does not support the use of jumbo frames for the software iSCSI initiator or for use with NFS datastores. At least, that’s my understanding.

So, feel free to tinker around with jumbo frames for IP-based storage, and when VMware adds official support for it in the future—I can’t imagine why they wouldn’t—then you’ll be able to hit the ground running with the configuration steps necessary to make it work.

A short while ago, I discussed that VMDKs on NFS may start out thin provisioned, but will lose that thin provisioned status over time. Operations like cloning and Storage VMotion will cause these thin provisioned disks to become thick (fully provisioned) disks, and you lose one of the benefits of running VMware on NFS.

Fortunately, if you’re running a NetApp storage system as your NFS server, you can preserve the thin provisioned status of these VMDKs by leveraging NetApp’s single file SnapRestore functionality. This article describes how that works.

There’s a couple of caveats here:

1. This technique only helps with making new VMs from an existing VM. SnapRestore won’t help preserve thin provisioned status after a Storage VMotion operation.
2. This isn’t integrated with VirtualCenter, so you won’t be able to take advantage of VirtualCenter’s integration with Sysprep and such.

As a result of #2 above, then, you’ll need to first prepare your source VM by running Sysprep inside the VM (assuming it is a Windows-based VM) and then allowing Sysprep to shut down the VM. Once that’s accomplished, then you can proceed.

The first step is to take a snapshot of the volume containing the already prepared VM:

snap create <vol-name> <snapshot-name>

Next, create a new VM in VirtualCenter, but do not create a virtual disk for the VM. This will create the VM configuration and associated files and the directory on the NFS datastore.

Third, run a SnapRestore operation to restore both the .vmdk file and the -flat.vmdk files. You have to restore both in order for this to work; keep in mind that the .vmdk file is just a header file and the -flat.vmdk is the actual disk file. The commands would look something like this:

snap restore -t file -s <snapshot-name> -r <new filename and path> <original filename and path>

As an example, let’s say you had a VM named template01 and you wanted to clone the disks for template01 to a new VM called newvm01, and these are stored on a volume called nfsvol. After you’ve run Sysprep on template01, taken the Snapshot and called it base_snapshot, and created newvm01 without a virtual disk, you’d run this command:

snap restore -t file -s base_snapshot -r /vol/nfsvol/newvm01/newvm01.vmdk /vol/nfsvol/template01/template01.vmdk

That would restore the .vmdk (header) file; then you’d restore the actual virtual disk file:

snap restore -t file -s base_snapshot -r /vol/nfsvol/newvm01/newvm01-flat.vmdk /vol/nfsvol/template01/template01-flat.vmdk

Once this process is complete—and it may take some time depending upon the size of the files being restored—you should see both the .vmdk and the -flat.vmdk files listed in the Datastore Browser.

“But wait a minute, Scott,” you say. “The -flat.vmdk no longer looks thin provisioned. You lied! This process doesn’t work.”

Indeed, it will look like it is no longer thin provisioned. Trust me; there’s one more step and then all will make sense. If you log into the ESX Server and open the .vmdk file in vi, you’ll see that it references the old file name of the -flat.vmdk. Edit that to reflect the new, restored file name, save your changes, and go back to the Datastore Broswer again. Refresh the display, and all should be well.

Why does the Datastore Browser work that way? Beats me. You’ll also find that running an “ls -la” on an NFS datastore from the Service Console will show you -flat.vmdk files that appear to be thick provisioned. The only way I’ve found to see if they are thin provisioned is to use the Datastore Browser. It’s a VMware thing, I suppose.

The last and final step is to edit the settings for the VM you created earlier and add the new virtual disk to that VM. Then you can boot up that VM and proceed with whatever customization steps, if any, are needed.

In the near future I plan to test another possible method of preserving the VMDK’s thin provisioned status, a method that is storage agnostic. Look for details of that testing here.

Here’s some thoughts on a variety of links that passed by me over the last couple of weeks. (Yes, I’ve been a bit lax in getting another Short Take published. Sorry.)

• Colleague Colin McNamara has written a good article about some of the challenges in integrating VMware into a Cisco network. He highlights something I’ve been saying for a while: a VMware implementation is more than just server virtualization; it affects servers, storage, networking, and security, and a good implementation requires addressing all of these areas as well as addressing things like staff organization and change management.
• Christofer Hoff started a good conversation about the performance implications of virtual security initiatives. It’s something many people are probably overlooking. After all, have you stopped to consider the additional processing power that running security products either inside the VMs, or at the hypervisor level, or both, will take from your CPU pool? I have a feeling that those high server consolidation ratios may not be so applicable when you factor in the security overhead.
• Per Duncan and Thomas, ESX Server 3.5 Update 1 will provide support for Microsoft Cluster Server (MSCS). Duncan also broke the news about the incorrect links for the update ESX ISOs.
• Massimo has initiated a discussion, picked up by the VMTN Blog, about the current state of high availability. I’m not a clustering expert, although I’ve setup my share of Microsoft clusters for SQL Server and Exchange Server. In my simplistic view, MSCS and VMware HA don’t really solve the same problem; MSCS is stateful (or mostly so), and VMware HA is stateless. Would you rather have a reasonably stateful failover for your Exchange Server, or would you rather have it rebooted? Stateful failover is not something that can be easily achieved in the virtual world right now, unless you bring MSCS into the virtual world; that, in turn, creates its own set of challenges. Continuous Availability, as demonstrated at VMworld 2007, will bring stateful failover to the virtual infrastructure.
• In the comments for the VMTN post about clustering vs. HA, reader “Matt” questions the use of NFS for VMware. In his linked article, he asks for a good white paper on why NFS instead of Fibre Channel. Well, I can’t provide a good white paper, but I can provide a couple useful articles, like this one or this one, to get started.
• David Marshall at VMblog has published parts one, two, and three of a three-part series on best practices for securing virtual networks. I haven’t had the opportunity to finish reading all three articles yet, but it looks like it’s avoided becoming an advertisement for Reflex Security.

Well, that wraps it up this time. Thanks for reading!

Here’s a quick list of a few of the articles that I currently have in progress:

• I’ve been looking at the recently-announced products from Altor Networks and have their VNSA (Virtual Network Security Analyzer) running in the lab right now. Look for a short post on the product and my initial thoughts. (One piece of advice I can offer to Altor right now: use the virtual appliance OVF format and save your customers a lot of trouble.)
• I’m going to have some hands-on time with a Xsigo I/O Director over the next few weeks, so look for an article on that product.
• Prodded to action by Nick’s comment in my article on thin provisioned VMDKs on NFS, I’ll have an article on using SnapRestore to clone VMs on NFS without losing the thin provisioned VMDKs.
• The Active Directory integration articles are getting a bit long in the tooth, so look for updates to both the Linux and Solaris integration instructions. (By the way, speaking of Solaris integration: I came across this article a short while ago.)

If anyone has any other topics they’d be interested in seeing me tackle, feel free to mention them in the comments below. I can’t make any promises, but I’ll certainly consider them!

UPDATE: The article on preserving thin provisioned VMDKs with SnapRestore is now available.

I’m relatively new to NetApp deduplication (formerly A-SIS), so this article won’t be an advanced treatise on NetApp deduplication or its deep inner workings. Instead, this is intended to be a quick guide to setting up NetApp deduplication for others, like myself, who may be familiar with Data ONTAP but not necessarily deduplication.

Obviously, the first step will be to ensure that your NetApp storage system is licensed for deduplication. As of March 10, NetApp made the NearStore option, which was a prerequisite for deduplication, free. Yes, you read that right: free. Since NearStore is a prerequisite, you’ll need to be sure to license that first:

license add <Code for NearStore>
license add <Code for Deduplication>

Once deduplication is licensed, then you can enable it on a per-volume basis using the “sis on” command:

sis on /vol/<volname>

Note, however, that the volume cannot exceed a certain size, based on the storage system model, in order for deduplication to work. These volume size limits are laid out in TR3505. Note that the volume must never have been any bigger than the size limits described, so this means you can’t size it down to the limits set forth and then run deduplication.

Once it’s running, you can check the status with:

sis status /vol/<volname>

After it’s finished running, you can see your space savings like this:

df -s /vol/<volname>

After running deduplication on a small NFS volume that housed only three VMs, the “df -s” command showed a space savings of 64%. That’s pretty impressive!

Moving forward, deduplication will run automatically every night at midnight, as shown by this command:

sis config /vol/<volname>

That should be enough to get most everyone started. Feel free to post comments or corrections below.

A niggling doubt about thin provisioned disks was placed in my head when I read Duncan’s article on a Storage VMotion problem; in that article, a statement is made that ESX Server 3.5 no longer supports thin provisioned disks. Intrigued by that comment, I started doing some digging to see if that was actually the case. I was unable to find any concrete statement one way or the other.

Some testing in my lab showed that with ESX Server 3.5, VMDKs are still thin provisioned by default when stored on an NFS datastore. So that put to rest the idea that thin provisioned disks had been abandoned, but now I was curious to follow up on the issue of how ESX Server handled cloning thin provisioned disks, as mentioned in Virtualization Short Take #1.

Additional testing showed that although the VMDKs are indeed thin provisioned at the beginning of their life, they won’t necessarily stay that way:

• Migration of the VMs files from one datastore to another, even if the destination datastore is also NFS, will cause the VMDKs to revert to thick (fully allocated) VMDKs.
• Clones made from the thin provisioned disks have thick provisioned VMDKs.
• A Storage VMotion operation will cause the disks to become fully allocated instead of thin provisioned.

This is a strong counterpoint to the arguments in favor of using NFS for your VMware storage in order to gain thin provisioned disks. In order to really take advantage of thin provisioned disks, every VM must be provisioned from scratch—no cloning within VirtualCenter—and you must give up Storage VMotion or cold migration of the VMDKs.

So far, I have not found any workaround for this behavior. If anyone knows of a workaround, please share it in the comments. (To be honest, I don’t really expect to find one.)

A friend of mine at Network Appliance was one of the presenters last year at VMworld 2007 for the now-famous presentation that showed a solution from Network Appliance where 100 VMs are created in just a couple of minutes. It’s great technology that is extremely useful in exactly those kinds of situations. I love it.

The video is so popular that it’s even been posted to YouTube. (By the way, did you know I’m on YouTube? My kids think that’s the greatest thing in the world, but I’m not so convinced.)

And, according to Manlio, it appears that they are showing off this kind of thing again at VMworld Europe 2008.

But it’s technology that’s not available yet.

Yep, that’s right. It’s not available yet. It’s based on new functionality, related to their existing FlexClone functionality (which I’ve blogged about before), that is due to be released very soon. Combine this new functionality with NFS on a NetApp storage system and you’ll be able to do exactly what NetApp is demonstrating. But not today…not until these new features are made available to the public.

That bothers me. I suppose it shouldn’t; I mean, you’ve got all sorts of vendors talking about their products and what their products can do when those products aren’t yet available. Microsoft Hyper-V is one example—it’s not available yet, won’t be until later this year, and yet Microsoft is showing it off. VMware is doing the same thing with the Continuous HA stuff they demo’ed at VMworld 2007. Likewise, VMware’s done the same thing this year with offline VDI and scalable virtual image technology.

So, if you’re thinking about a huge VDI deployment and planning on putting that on NetApp storage, that’s fine because there are plenty of other reasons to use Network Appliance—deduplication, anyone? But don’t plan on being able to take advantage of some of this highly touted functionality until it is publicly released.

UPDATE: Another colleague of mine at NetApp wrote me to clarify that the file-level cloning functionality demonstrated in the video is not, technically speaking, related to FlexClone functionality since FlexClone operates on a per-volume basis. I might argue that they both appear to exploit the same underlying functionality in WAFL, but I don’t know that for certain and at that point we’re splitting hairs anyway.

As sometimes happens, I’ve been collecting a variety of virtualization-related links that, while interesting, don’t necessarily warrant a full blog posting by themselves.  Since I don’t want to just copy someone else’s content and not provide any value of my own, I’ve decided to start irregularly publishing a “Virtualization Short Take”.  Each of these will just be a few links with a brief commentary or thought attached.  Perhaps these will lead to broader and more active discussions around some of these topics.

Without further delay, then, here’s the first-ever Virtualization Short Take:

• Via Duncan, a new and undocumented feature in VCB’s config.js file that allows for non-quiesced snapshots of virtual machines.  I’d be interested in more discussion around why some workloads might be better served by non-quiesced snapshots, so if anyone has some insight please speak up.
• Also via Duncan, fixes for crashes caused by installing the Converter plugin to VirtualCenter.  Two solutions are available; try this one, then try this approach.
• Via Thomas, it looks like VMware has published a storage performance white paper comparing Fibre Channel, hardware iSCSI, software iSCSI, and NFS.  I just finished reviewing the document myself, and plan to go back and look at it again more thoroughly.  It’s useful information for virtualization architects or engineers responsible for designing storage solutions for VMware.
• In case anyone’s interested in creating their own bootable ESX 3i USB drive, here’s more information.  This is something I need to take a closer look at myself, so when I have the chance to run through these instructions I’ll try to post some feedback on how well they worked.
• Based on some interaction with a customer yesterday, it looks as if VirtualCenter 2.5 may require the Power User role to be directly assigned to the Hosts & Clusters object in order for users to be able to attach local (client-side) ISO files to a VM.  Has anyone else seen this behavior?  I’m going to try to recreate the problem myself, but if you’ve seen this please speak up in the comments.
• I’ve also recently received word from a friend that the thin-provisioned disks VMware uses by default on an NFS datastore may not be honored when cloning VMs from a template.  Again, I plan to test this myself, but if anyone else out there has seen this behavior I’d love to hear about it.

As always, I’d love to hear your thoughts, so feel free to add your voice in the comments below.