NetApp

What I had hoped to be able to publish today would be an article describing how to configure and use ESX’s software iSCSI initiator as a failover path for Fibre Channel, so that if the Fibre Channel fabric completely failed VM traffic would automatically failover to software iSCSI. I thought that this would be a great, low-cost way to add another layer of redundancy to your VMware ESX environment.

Unfortunately, I can’t make it work. Here’s the setup I’ve been using for testing:

• A 200GB LUN visible to ESX over both Fibre Channel (FC) and software iSCSI
• A VM, stored on this LUN, running Windows Server 2003 R2

Initial tests led me to believe that it would indeed work. I verified that both the FC path as well as the iSCSI path were listed as separate paths for the same LUN. Without placing any load on the VM, I pulled the FC connection from the back of the server. The VM stayed up, and I was able to browse the local hard drive inside the VM. Network connectivity remained active. And the “Manage Paths” dialog box even showed the FC connection as “Dead” and the iSCSI connection as On/Active. Given that information, it seemed like all was good.

Determined to verify that it was working as I expected, I trotted out a copy of IOmeter and tried to repeat the tests. This time around, though, the tests did not go quite so well. IOmeter showed that disk throughput stopped, and the VI Client locked up. I repeated this set of tests a couple of times, and each time—while IOmeter was running—I ran into issues.

Based on these results, I’m inclined to say that one of two things is true. Either:

1. I did something very, very wrong; or
2. ESX isn’t quite right to support automatic failover between FC and software iSCSI.

Has anyone else tried this, or am I the only one? If you have tried it, did it work? If so, what steps did you have to take—if any—to make it work properly?

Building on my earlier article on setting up NetApp deduplication, I wanted to follow up with some information on using NetApp deduplication with block storage (LUNs presented via Fibre Channel or iSCSI).

For the most part, using NetApp deduplication with block storage is a lot like I described earlier:

• You (obviously) still need the NearStore and deduplication (A-SIS) licenses installed on the controller(s).
• You will still turn deduplication on using the “sis on” command for the FlexVol containing the LUNs.
• Limitations on the size of the FlexVol still apply.
• You use the “sis status” command to check on the status of deduplication, and the “sis config” command to see the deduplication schedule.

OK, so what’s different? Well, it has to do with how LUNs are provisioned on a NetApp storage system. I’ve blogged before about managing LUN space requirements on a NetApp, and about using LUN clones vs. FlexClones. That second article, in particular, really goes into detail on how LUNs are implemented on top of NetApp’s file system, WAFL. Since LUNs are represented by WAFL as a single file, they are also normally “space reserved,” meaning that the maximum size of the LUN is allocated at the time of creation. If you create a 50GB LUN, then Data ONTAP creates a 50GB file right away. (For readers out there who are well-versed in NetApp storage, I know that’s a bit of a simplification, but bear with me.)

What does this have to do with deduplication? Great question. If the LUN is space reserved—meaning that the maximum size of the LUN is allocated up front and remains allocated to the LUN—then the file that represents the LUN won’t ever decrease in size to reflect deduplication savings, and deduplication therefore does you absolutely no good whatsoever. This is not to say that deduplication doesn’t work, just that it won’t help you at all.

Fortunately, there’s an easy fix for this. When creating the LUN, simply uncheck the box marked “Space Reserved” and allow Data ONTAP to allocate space to the LUN out of the containing FlexVol on an as-needed basis. Because the file that represents the LUN can grow in size, it can also shrink in size, and deduplication will cause the file that represents the LUN to decrease in size. This then allows you to provision additional LUNs from the same FlexVol to take advantage of the space savings resulting from deduplication.

I know that seems a bit confusing; I’ll probably post another article with some more in-depth discussions of the details. (Either that, or I’ll encourage my NetApp readers to chime in below in the comments.)

So, in summary, when using NetApp deduplication with block storage:

• you’ll setup and configure deduplication on the FlexVol containing your LUN(s) just like described in my earlier article;
• you’ll uncheck the “Space Reserved” checkbox when creating the LUNs to be deduplicated;
• you won’t see the space savings from the host’s perspective and therefore can’t store more data in that LUN than the size of the LUN; but
• you will be able to provision additional LUNs in that same FlexVol that can be presented back to host for additional storage.

I hope this helps clarify some of the questions or issues surrounding the use of NetApp deduplication with block storage. Feel free to add information, experiences with deduplication and block storage, or ask additional questions in the comments below.

UPDATE: There are some additional considerations about how to provision LUNs along with NetApp deduplication that warrant a more in-depth discussion. Look for a follow-up post within the next few days.

With the release of VMware Infrastructure version 3.5, VMware added support for jumbo frames. Although the documentation states that jumbo frames “are not supported for NAS and iSCSI traffic”, jumbo frames for NFS and iSCSI does actually work. Here’s some information on getting it working.

How I Tested

Keep in mind that this is not an “officially supported” configuration (see the section on the “Official” Support Statement below), so use at your own risk. I will not be held responsible if you blow up your production environment trying to make jumbo frames work.

Here’s how I tested the use of jumbo frames for software iSCSI and NFS datastores:

• For the physical switch infrastructure, I used a Cisco Catalyst 3560G running Cisco IOS version 12.2(25)SEB4.
• For the physical server hardware, I used a group of HP ProLiant DL385 G2 servers with dual-core AMD Opteron processors and a quad-port PCIe Intel Gigabit Ethernet NIC.
• For the storage system, I used a NetApp FAS940 running Data ONTAP 7.2.4.

The exact commands and/or procedures may be different for you depending upon the hardware and/or software versions that you’re running in your environment. Keep that in mind.

Configuring the Physical Switch

Fortunately for me, the Cisco Catalyst 3560G does indeed support jumbo frames. (Naturally, you’ll want to ensure that your switch supports jumbo frames.) Jumbo frames are not, however, enabled by default; they must be enabled using the following command in global configuration mode:

system mtu jumbo 9000

Note that 9000 bytes seems to be the generally accepted size for jumbo frames, so that’s what I used.

After running this command, you must reboot the switch. The change doesn’t take effect until a reload. Fortunately, IOS reminds you of this after you enter the command. Once the switch has rebooted, you can verify the MTU setting with this command:

show system mtu

This should report that the system jumbo MTU size is 9000 bytes, confirming that the switch is ready for jumbo frames. Now we’re prepared to configure the storage system.

Configuring the Storage System

Using FilerView, increasing the MTU on the appropriate network interfaces to 9000 bytes is as simple as going to Network > Manage Interfaces and then clicking the Modify link for the interface to be changed. Set the “MTU size” to 9000 (from the default of 1500), click Apply, and you’re ready to roll.

You can verify the settings in FilerView using Network > Manage Interfaces > Show All Interface Details, or by using the “ifconfig -a” command from a Data ONTAP command prompt.

Configuring ESX Server

There is no GUI in VirtualCenter for configuring jumbo frames; all of the configuration must be done from a command line on the ESX server itself. There are two basic steps:

1. Configure the MTU on the vSwitch.
2. Create a VMkernel interface with the correct MTU.

First, we need to set the MTU for the vSwitch. This is pretty easily accomplished using esxcfg-vswitch:

esxcfg-vswitch -m 9000 vSwitch1

A quick run of “esxcfg-vswitch -l” (that’s a lowercase L) will show the vSwitch’s MTU is now 9000; in addition, “esxcfg-nics -l” (again, a lowercase L) will show the MTU for the NICs linked to that vSwitch are now set to 9000 as well.

Second, we need to create a VMkernel interface. This step is a bit more complicated, because we need to have a port group in place already, and that port group needs to be on the vSwitch whose MTU we set previously:

esxcfg-vmknic -a -i 172.16.1.1 -n 255.255.0.0 -m 9000 IPStorage

This creates a port group called IPStorage on vSwitch1—the vSwitch whose MTU was previously set to 9000—and then creates a VMkernel port with an MTU of 9000 on that port group. Be sure to use an IP address that is appropriate for your network when creating the VMkernel interface.

To test that everything is working so far, use the vmkping command:

vmkping -s 9000 172.16.1.200

Clearly, you’ll want to substitute the IP address of your storage system in that command.

That’s it! From here you should be able to easily add an NFS datastore or connect to an iSCSI LUN using jumbo frames from the ESX server.

“Official” Support Statement

Officially, jumbo frames are only supported by VMware for use by virtual machines. Technically, VMware does not support the use of jumbo frames for the software iSCSI initiator or for use with NFS datastores. At least, that’s my understanding.

So, feel free to tinker around with jumbo frames for IP-based storage, and when VMware adds official support for it in the future—I can’t imagine why they wouldn’t—then you’ll be able to hit the ground running with the configuration steps necessary to make it work.

VMware’s done a great job of rolling together the VMware news and views with their VMware blog aggregator, Planet V12n. Does anyone know of something equivalent for NetApp? Where is “Planet NetApp”?

A short while ago, I discussed that VMDKs on NFS may start out thin provisioned, but will lose that thin provisioned status over time. Operations like cloning and Storage VMotion will cause these thin provisioned disks to become thick (fully provisioned) disks, and you lose one of the benefits of running VMware on NFS.

Fortunately, if you’re running a NetApp storage system as your NFS server, you can preserve the thin provisioned status of these VMDKs by leveraging NetApp’s single file SnapRestore functionality. This article describes how that works.

There’s a couple of caveats here:

1. This technique only helps with making new VMs from an existing VM. SnapRestore won’t help preserve thin provisioned status after a Storage VMotion operation.
2. This isn’t integrated with VirtualCenter, so you won’t be able to take advantage of VirtualCenter’s integration with Sysprep and such.

As a result of #2 above, then, you’ll need to first prepare your source VM by running Sysprep inside the VM (assuming it is a Windows-based VM) and then allowing Sysprep to shut down the VM. Once that’s accomplished, then you can proceed.

The first step is to take a snapshot of the volume containing the already prepared VM:

snap create <vol-name> <snapshot-name>

Next, create a new VM in VirtualCenter, but do not create a virtual disk for the VM. This will create the VM configuration and associated files and the directory on the NFS datastore.

Third, run a SnapRestore operation to restore both the .vmdk file and the -flat.vmdk files. You have to restore both in order for this to work; keep in mind that the .vmdk file is just a header file and the -flat.vmdk is the actual disk file. The commands would look something like this:

snap restore -t file -s <snapshot-name> -r <new filename and path> <original filename and path>

As an example, let’s say you had a VM named template01 and you wanted to clone the disks for template01 to a new VM called newvm01, and these are stored on a volume called nfsvol. After you’ve run Sysprep on template01, taken the Snapshot and called it base_snapshot, and created newvm01 without a virtual disk, you’d run this command:

snap restore -t file -s base_snapshot -r /vol/nfsvol/newvm01/newvm01.vmdk /vol/nfsvol/template01/template01.vmdk

That would restore the .vmdk (header) file; then you’d restore the actual virtual disk file:

snap restore -t file -s base_snapshot -r /vol/nfsvol/newvm01/newvm01-flat.vmdk /vol/nfsvol/template01/template01-flat.vmdk

Once this process is complete—and it may take some time depending upon the size of the files being restored—you should see both the .vmdk and the -flat.vmdk files listed in the Datastore Browser.

“But wait a minute, Scott,” you say. “The -flat.vmdk no longer looks thin provisioned. You lied! This process doesn’t work.”

Indeed, it will look like it is no longer thin provisioned. Trust me; there’s one more step and then all will make sense. If you log into the ESX Server and open the .vmdk file in vi, you’ll see that it references the old file name of the -flat.vmdk. Edit that to reflect the new, restored file name, save your changes, and go back to the Datastore Broswer again. Refresh the display, and all should be well.

Why does the Datastore Browser work that way? Beats me. You’ll also find that running an “ls -la” on an NFS datastore from the Service Console will show you -flat.vmdk files that appear to be thick provisioned. The only way I’ve found to see if they are thin provisioned is to use the Datastore Browser. It’s a VMware thing, I suppose.

The last and final step is to edit the settings for the VM you created earlier and add the new virtual disk to that VM. Then you can boot up that VM and proceed with whatever customization steps, if any, are needed.

In the near future I plan to test another possible method of preserving the VMDK’s thin provisioned status, a method that is storage agnostic. Look for details of that testing here.

Here’s a quick list of a few of the articles that I currently have in progress:

• I’ve been looking at the recently-announced products from Altor Networks and have their VNSA (Virtual Network Security Analyzer) running in the lab right now. Look for a short post on the product and my initial thoughts. (One piece of advice I can offer to Altor right now: use the virtual appliance OVF format and save your customers a lot of trouble.)
• I’m going to have some hands-on time with a Xsigo I/O Director over the next few weeks, so look for an article on that product.
• Prodded to action by Nick’s comment in my article on thin provisioned VMDKs on NFS, I’ll have an article on using SnapRestore to clone VMs on NFS without losing the thin provisioned VMDKs.
• The Active Directory integration articles are getting a bit long in the tooth, so look for updates to both the Linux and Solaris integration instructions. (By the way, speaking of Solaris integration: I came across this article a short while ago.)

If anyone has any other topics they’d be interested in seeing me tackle, feel free to mention them in the comments below. I can’t make any promises, but I’ll certainly consider them!

UPDATE: The article on preserving thin provisioned VMDKs with SnapRestore is now available.

For readers that didn’t already know this, Snapshots on NetApp-hosted CIFS shares are automatically and transparently recognized by Windows versions that support the “Previous Versions” tab. I believe this functionality is native on Windows Server 2003 and an add-on for Windows XP, but I’m not 100% certain. Either way, this means that users can easily recover files from NetApp Snapshots from within the Windows GUI.

Here’s a screenshot of the Previous Versions support on an out-of-the-box server running Windows Server 2003 R2:

As you can see in the screenshot, the Previous Versions tab automatically shows the NetApp Snapshots. No configuration is required; this is after a vanilla installation of Windows Server 2003 R2 and all applicable updates from Windows Update.

To NetApp veterans, this is nothing new, but I thought some new NetApp users out there might find information about this functionality and integration useful.

I’m relatively new to NetApp deduplication (formerly A-SIS), so this article won’t be an advanced treatise on NetApp deduplication or its deep inner workings. Instead, this is intended to be a quick guide to setting up NetApp deduplication for others, like myself, who may be familiar with Data ONTAP but not necessarily deduplication.

Obviously, the first step will be to ensure that your NetApp storage system is licensed for deduplication. As of March 10, NetApp made the NearStore option, which was a prerequisite for deduplication, free. Yes, you read that right: free. Since NearStore is a prerequisite, you’ll need to be sure to license that first:

license add <Code for NearStore>
license add <Code for Deduplication>

Once deduplication is licensed, then you can enable it on a per-volume basis using the “sis on” command:

sis on /vol/<volname>

Note, however, that the volume cannot exceed a certain size, based on the storage system model, in order for deduplication to work. These volume size limits are laid out in TR3505. Note that the volume must never have been any bigger than the size limits described, so this means you can’t size it down to the limits set forth and then run deduplication.

Once it’s running, you can check the status with:

sis status /vol/<volname>

After it’s finished running, you can see your space savings like this:

df -s /vol/<volname>

After running deduplication on a small NFS volume that housed only three VMs, the “df -s” command showed a space savings of 64%. That’s pretty impressive!

Moving forward, deduplication will run automatically every night at midnight, as shown by this command:

sis config /vol/<volname>

That should be enough to get most everyone started. Feel free to post comments or corrections below.

If I had one complaint about being a Mac OS X user, it would be that there is no Visio equivalent on the Mac. Yes, OmniGraffle is an awesome program; this is especially true for version 5, which can open native Visio drawing (VSD) and shape (VSS) files. That one new feature was, to me, worth the upgrade price.

<aside>Note to OmniGroup developers: It would be really, really, REALLY helpful to do something with the Windows Metafile images that Visio uses instead of just replacing them with a gray rectangle. Do you have something up your sleeve to help with that? Perhaps a trick of which I am not aware?</aside>

OK, so I suppose a more accurate complaint is about a lack of shapes for OmniGraffle more so than anything else. Fortunately, I came across this link last night. Now, my need for high-quality NetApp shapes for OmniGraffle diagrams has been almost entirely satisfied. To the person that created these shapes and posted them, you have my sincere thanks!

Anyone else have any little gems like this they’d like to share with other readers?

Once again, here’s my take a few virtualization-related stories that have passed through my computer in the last few days:

• OK, this first one isn’t technically related to virtualization, but it was too good to pass up. Is there anyone besides me and The Register who thinks NetApp’s new logo is…um…well, not as good as the previous one?
• A new blog war is brewing between VMware and Citrix, and this time I had nothing to do with it: VMware apparently launched the first volley in discussing the value of ESX Server’s memory overcommitment and page sharing functionality; Citrix’s Roger Klorese then responded and Simon Crosby chimed in as well. I would completely agree with Roger’s and Simon’s comments, except for this one statement in Eric’s original post:
  

  We created and powered on 512MB Windows XP VMs running a light workload [emphasis mine] and kept adding them until the server couldn’t take any more.

  Since Eric stated the parameters of the test involved lightly loaded workstations, Roger’s comments about heavy workloads don’t apply. Besides, any engineer worth his/her weight isn’t going to overcommit a production workload like that, and this analysis shows that some overcommitment can produce notable financial results.

• CIO Magazine recently published a list of 10 virtualization risks hiding in your company. It’s a pretty interesting list, although it’s worthwhile to note that this list was produced by a VP of Marketing for Embotics and therefore is heavily slanted toward the risks that his company’s products can help mitigate.
• This is interesting and novel, but that’s about it. (UPDATE: The creator of the 37migrations VI plugin, Schley Andrew Kutz, wrote me to state that there is no point in 37migrations; it’s just for fun. So stop trying to find a deeper meaning in it, OK?)
• There’s apparently a problem with using Sysprep in VirtualCenter 2.5 with Windows Server 2003 SP2. A Microsoft hotfix is available.
• Speaking of NetApp, they’ve been generating some buzz around their SnapManager for Virtual Infrastructure (SMVI) product, yet another unreleased product. I echo Duncan’s thoughts about the VC plugin!
• Gabe shares some information he’s gathered about VMsafe, the recently announced security APIs from VMware.
• Alessandro shares his thoughts about Microsoft’s virtualization strategy following the announcement of Microsoft’s purchase of Kidaro. My question is this: was VMware’s announcement of offline VDI functionality at VMworld Europe 2008 because they had an inkling of Microsoft’s moves, or is Microsoft’s purchase a result of VMware’s announcement?

That’s it for today. Join in the discussion by adding your 2 cents in the comments below!