blog.scottlowe.org

The recent couple of articles I wrote about using NetApp deduplication—in particular, the article on using NetApp deduplication with block storage—have raised some questions that are probably worth addressing. Although NetApp deduplication works just fine with block-based storage, there are some considerations with regards to how the LUNs should be provisioned when deduplication will come into play.

Fortunately for me, someone at NetApp decided that it would be a good idea to document the five basic configurations of using NetApp deduplication with block storage. As seen in this comment to an earlier article, Larry Freeman points out this document on the NetApp Communities site (has anyone else noticed the similarity between VMware Communities and NetApp Communities?) that outlines the 5 basic configurations and where the freed blocks go in each configuration. Excellent—that saves me some work!

The most common configurations I’ve seen are configurations D (LUN not space reserved, space guarantee set to Volume) and E (LUN not space reserved, space guarantee set to None). Customers like to see the LUNs “shrink” after deduplication, and this is the only way to make that happen.

The only things we need now are for NetApp to a) remove the volume size limitations and b) get us deduplication at the aggregate level. Then we’d really be set!

Building on my earlier article on setting up NetApp deduplication, I wanted to follow up with some information on using NetApp deduplication with block storage (LUNs presented via Fibre Channel or iSCSI).

For the most part, using NetApp deduplication with block storage is a lot like I described earlier:

• You (obviously) still need the NearStore and deduplication (A-SIS) licenses installed on the controller(s).
• You will still turn deduplication on using the “sis on” command for the FlexVol containing the LUNs.
• Limitations on the size of the FlexVol still apply.
• You use the “sis status” command to check on the status of deduplication, and the “sis config” command to see the deduplication schedule.

OK, so what’s different? Well, it has to do with how LUNs are provisioned on a NetApp storage system. I’ve blogged before about managing LUN space requirements on a NetApp, and about using LUN clones vs. FlexClones. That second article, in particular, really goes into detail on how LUNs are implemented on top of NetApp’s file system, WAFL. Since LUNs are represented by WAFL as a single file, they are also normally “space reserved,” meaning that the maximum size of the LUN is allocated at the time of creation. If you create a 50GB LUN, then Data ONTAP creates a 50GB file right away. (For readers out there who are well-versed in NetApp storage, I know that’s a bit of a simplification, but bear with me.)

What does this have to do with deduplication? Great question. If the LUN is space reserved—meaning that the maximum size of the LUN is allocated up front and remains allocated to the LUN—then the file that represents the LUN won’t ever decrease in size to reflect deduplication savings, and deduplication therefore does you absolutely no good whatsoever. This is not to say that deduplication doesn’t work, just that it won’t help you at all.

Fortunately, there’s an easy fix for this. When creating the LUN, simply uncheck the box marked “Space Reserved” and allow Data ONTAP to allocate space to the LUN out of the containing FlexVol on an as-needed basis. Because the file that represents the LUN can grow in size, it can also shrink in size, and deduplication will cause the file that represents the LUN to decrease in size. This then allows you to provision additional LUNs from the same FlexVol to take advantage of the space savings resulting from deduplication.

I know that seems a bit confusing; I’ll probably post another article with some more in-depth discussions of the details. (Either that, or I’ll encourage my NetApp readers to chime in below in the comments.)

So, in summary, when using NetApp deduplication with block storage:

• you’ll setup and configure deduplication on the FlexVol containing your LUN(s) just like described in my earlier article;
• you’ll uncheck the “Space Reserved” checkbox when creating the LUNs to be deduplicated;
• you won’t see the space savings from the host’s perspective and therefore can’t store more data in that LUN than the size of the LUN; but
• you will be able to provision additional LUNs in that same FlexVol that can be presented back to host for additional storage.

I hope this helps clarify some of the questions or issues surrounding the use of NetApp deduplication with block storage. Feel free to add information, experiences with deduplication and block storage, or ask additional questions in the comments below.

UPDATE: There are some additional considerations about how to provision LUNs along with NetApp deduplication that warrant a more in-depth discussion. Look for a follow-up post within the next few days.

VMware’s done a great job of rolling together the VMware news and views with their VMware blog aggregator, Planet V12n. Does anyone know of something equivalent for NetApp? Where is “Planet NetApp”?

I’m relatively new to NetApp deduplication (formerly A-SIS), so this article won’t be an advanced treatise on NetApp deduplication or its deep inner workings. Instead, this is intended to be a quick guide to setting up NetApp deduplication for others, like myself, who may be familiar with Data ONTAP but not necessarily deduplication.

Obviously, the first step will be to ensure that your NetApp storage system is licensed for deduplication. As of March 10, NetApp made the NearStore option, which was a prerequisite for deduplication, free. Yes, you read that right: free. Since NearStore is a prerequisite, you’ll need to be sure to license that first:

license add <Code for NearStore>
license add <Code for Deduplication>

Once deduplication is licensed, then you can enable it on a per-volume basis using the “sis on” command:

sis on /vol/<volname>

Note, however, that the volume cannot exceed a certain size, based on the storage system model, in order for deduplication to work. These volume size limits are laid out in TR3505. Note that the volume must never have been any bigger than the size limits described, so this means you can’t size it down to the limits set forth and then run deduplication.

Once it’s running, you can check the status with:

sis status /vol/<volname>

After it’s finished running, you can see your space savings like this:

df -s /vol/<volname>

After running deduplication on a small NFS volume that housed only three VMs, the “df -s” command showed a space savings of 64%. That’s pretty impressive!

Moving forward, deduplication will run automatically every night at midnight, as shown by this command:

sis config /vol/<volname>

That should be enough to get most everyone started. Feel free to post comments or corrections below.

Via Nick, it looks like the next version of Citrix XenServer (version 4.1, currently in beta) will have a new feature called Storage Delivery Services that is designed to take full advantage of some of the features of NetApp storage systems. Quoting from the Citrix datasheet:

The adapter uses native NetApp APIs to implement XenAPI storage operations that provide access to hardware provisioning and snapshotting directly, without the proprietary layer that complicates management in other products.

To put that in normal English that people can understand, Citrix has written an “adapter” that talks directly to Data ONTAP to enable XenServer to take full advantage of functionality such as Snapshots, FlexVols, and FlexClones directly from Xen management tools. This allows Xen management tools that use the XenAPI to provision VMs using cloning functionality in the storage hardware, for example. And it can do that without some of the drawbacks that come from the use of storage system functionality to provision VMs (I describe some advantages and disadvantages here and here).

This is pretty significant, in my mind, and definitely gives Citrix XenServer a hand up on VMware who, so far, has continued to remain storage agnostic. Of course, by remaining storage agnostic, they’ve also chosen to forgo the benefits of tying directly into the storage hardware.

Virtualization is about using your hardware more effectively. That should include your storage as well.

In my previous article about using NetApp multi-mode VIFs with Cisco switches, I mentioned that you could—at that time—only use 802.3ad static link aggregation:

Be aware that Data ONTAP’s multi-mode VIFs are only compatible with static 802.3ad link aggregation; you can’t use PAgP (Cisco proprietary protocol).  I would assume dynamic LACP is also incompatible.  For this reason we used the “channel-group 1 mode on” statement instead of something like “channel-group 1 mode desirable”.

I recently got some feedback from a NetApp SE in my area; this SE informed me that Link Aggregation Control Protocol (LACP, part of the IEEE 802.3ad specification) is indeed supported with Data ONTAP version 7.2.  This KB article on the NetApp NOW site (login required) indicates that ONTAP 7.2.1 is required in order to use a LACP VIF.

There are a couple important requirements to note; these are laid out in the referenced KB article:

1. Dynamic multimode VIFs should use IP address-based load balancing.  This means that the Cisco switch or the channel group must also use IP address-based load balancing.
2. Dynamic multimode VIFs must be first-level VIFs.  This makes sense; LACP is a Layer 2 protocol, so layering a LACP VIF on top of other VIFs just doesn’t work.

To create the dynamic multimode VIF on the Data ONTAP side, the command is pretty simple:

vif create lacp <vif name> -b ip {interface list}

On the Cisco side, the commands are very similar:

s3(config)#int port-channel1
s3(config-if)#description LACP multimode VIF for netapp1
s3(config-if)#int gi0/23
s3(config-if)#channel-protocol lacp
s3(config-if)#channel-group 1 mode active

These commands would be repeated for all physical ports that should be included in the LACP bundle.  Note the differences from the earlier commands in the previous article; here we use “channel-group 1 mode active” instead of “channel-group 1 mode on”.  We also added the “channel-protocol lacp” command.

Together, these commands will establish a LACP-based link aggregate between a NetApp storage system running Data ONTAP version 7.2.1 or higher and a Cisco IOS-based switch.

Thanks to Jeff, our NetApp SE, for providing the updated information.

If you’ve worked with Network Appliance storage before, you’re probably already familiar with the idea of snap reserve (storage space set aside to accommodate for Snapshots) and fractional reserve (used with LUNs).  I’m going to hold the in-depth discussion of why you need snap reserve and fractional reserve for a different day, but I did want to pass on these commands that were shared with me by a colleague of mine.  These Data ONTAP commands, available with Data ONTAP 7.2 or later (some commands are available in Data ONTAP 7.1), will help you manage the space requirements for LUNs on a NetApp storage area network (SAN).

I’ll try to explain the commands along the way, but I would recommend you review the documentation available from the NOW site for more complete information.

vol options <volname> fractional_reserve 0

This command sets the fractional reserve to zero percent, down from the default of 100 percent.  Note that fractional reserve only applies to LUNs, not to NAS storage presented via CIFS or NFS.

snap autodelete <volname> trigger snap_reserve

This sets the trigger at which Data ONTAP will begin deleting Snapshots.  In this case, Snapshots will start getting deleted when the snap reserve for the volume gets nearly full.  The current size of the snap reserve can be viewed for a particular volume with the “snap reserve <volname>” command.

snap autodelete <volname> defer_delete none

This command instructs Data ONTAP not to exhibit any preference in the types of Snapshots that are deleted.  Options for this command include “user_created” (delete user-created Snapshot copies last) or “prefix” (Snapshot copies with a specified prefix string).

snap autodelete <volname> target_free_space 10

With this setting in place, Snapshots will be deleted until there is 10% free space in the volume.

snap autodelete <volname> on

Now that the Snapshot autodelete options have been configured, this command will actually turn the functionality on.

vol options <volname> try_first snap_delete

When a FlexVol runs into an issue with space, this option tells Data ONTAP to first try to delete Snapshots in order to free up space.  This command works in conjunction with the next command:

vol autosize <volname> on

This enables Data ONTAP to automatically grow the size of a FlexVol if the need arises.  This command works hand-in-hand with the previous command; Data ONTAP will first try to delete Snapshots to free up space, then grow the FlexVol according to the autosize configuration options.  Between these two options—Snapshot autodelete and volume autogrow—you can reduce the fractional reserve from the default of 100 and still make sure that you don’t run into problems taking Snapshots of your LUNs.

If you have a NOW login, you can get more information on Snapshot autodelete here; more information on volume autogrow is available here.  Be aware that SnapDrive may require different settings in order to accommodate its functionality, as it moves LUN management out of the storage system and onto the host.  Finally, the values presented here are only examples; be sure to use values that are appropriate for your environment.

Credit for compiling this list goes to my colleague Chauncey Willard.  Good work!

Network Appliance storage systems support the use of virtual interfaces (VIFs) to provide link redundancy and improved network throughput.  Two types of VIFs are available:

• Single-mode VIFs act like a fault tolerant team and will fail traffic over to a standby link when the active link goes down.
• Multi-mode VIFs act like a group of links providing aggregate bandwidth as well as link redundancy.

Single-mode VIFs are great for fault tolerance, but the storage system isn’t leveraging all the links.  It’s “active-passive” arrangement in which only one of the links is passing traffic while the other link is idle.  No switch support is required for this configuration.

Multi-mode VIFs, on the other hand, allow for both greater bandwidth utilization as well as fault tolerance.  Traffic will be distributed across all the links in the VIF (typically based on IP address), and if one link fails the traffic is redistributed across the remaining links.  However, this configuration requires support on the switch.  In this article, we’re going to look at configuring a Cisco Catalyst 3560 switch to do link aggregation with a NetApp storage system running Data ONTAP 7.1.1.1.

To configure the switch, we’ll use the following commands (these are entered in global configuration mode on the switch):

s3(config)#int port-channel1
s3(config-if)#description Multi-mode VIF for netapp1
s3(config-if)#int gi0/23
s3(config-if)#channel-group 1 mode on
s3(config-if)#int gi0/24
s3(config-if)#channel-group 1 mode on

This creates the port-channel1 interface (you may need to increment that number, i.e., use port-channel2 or port-channel3, if you already have existing link aggregates configured) and adds interfaces GigabitEthernet0/23 and GigabitEthernet0/24 to the link aggregate.  If you do have to use a different link aggregate interface, be sure the number of the interface (“int port-channel4”) matches the number of the channel-group specified on the member interfaces (“channel-group 4 mode on”).  This seems obvious, but it’s worth mentioning nevertheless.

Be aware that Data ONTAP’s multi-mode VIFs are only compatible with static 802.3ad link aggregation; you can’t use PAgP (Cisco proprietary protocol).  I would assume dynamic LACP is also incompatible.  For this reason we used the “channel-group 1 mode on” statement instead of something like “channel-group 1 mode desirable”.

By default, many Cisco switches default to MAC address-based load balancing across the links, whereas NetApp defaults to IP address-based load balancing.  To see the switch’s current load balancing configuration, use this command in privileged mode:

s3#show etherchannel load-balance

To change the switch’s load balancing algorith to a mode compatible with NetApp’s, use one of the following command in global configuration mode (note that changing it affects the entire switch; you can’t change it for a single port-channel individually):

s3(config)#port-channel load-balance src-dst-ip

Once the switch is configured, then we can proceed with configuring the NetApp storage system.  The following commands will create the the multi-mode VIF (this can also be done via the FilerView GUI):

netapp1>vif create multi vif0 e6d e7d
netapp1>ifconfig vif0 172.31.254.10 netmask 255.255.255.0
netapp1>ifconfig vif0 up

This creates the VIF with interfaces e6d and e7d as members, plumbs it with an IP address, and brings it up.  Running the command “vif status vif0” now will return the following results:

default: transmit ‘IP Load balancing’, VIF Type ‘multi_mode’, fail ‘log’
vif0: 2 links, transmit ‘IP Load balancing’, VIF Type ‘multi-mode’ fail ‘default’
 
VIF Status Up Addr_set
up:
e6d: state up, since 05Oct2001 17:17:15 (05:23:05)
mediatype: auto-1000t-fd-up
flags: enabled
input packets 2000, input bytes 12800
output packets 173, output bytes 1345
up indications 1, broken indications 0
drops (if) 0, drops (link) 0
indication: up at boot
consecutive 3, transitions 1
e7d: state up, since 05Oct2001 17:18:03 (00:10:03)
mediatype: auto-1000t-fd-up
flags: enabled
input packets 134, input bytes 987
output packets 20, output bytes 156
up indications 1, broken indications 0
drops (if) 0, drops (link) 0
indication: broken

Note the ‘IP Load balancing’ algorithm stated in the output; this is why the switch’s load-balancing mechanism should be changed to match.

At this point, the links should be up between the Cisco switch and the NetApp storage system, and traffic should be passing to and from the storage system without any problems.  To test the fault tolerance, we can pull one of the links in VIF; traffic should continue to flow with very little, if any, interruption.  And while traffic from a single client to the NetApp won’t see a significant increase in throughput, the overall throughput of multiple separate clients to the NetApp should improve with multiple links in the VIF.

More information, including additional Cisco configs, is available here.

To be honest, there are times when using Mac OS X can be difficult.  With the overwhelming installed base of Windows and the attention of the media darling Linux, vendors will often provide instructions on how to do something with Windows or Linux, but not from Mac OS X.  Sometimes the Linux instructions work, but many times they don’t, and the Windows instructions typically involve running some sort of Windows executable that won’t, of course, run on Mac OS X.  What is one to do, then?

This is situation in which I found myself last week while trying to upgrade a Network Appliance storage system (an old F840) in the lab.  I wanted to upgrade the F840 to Data ONTAP 7.1.2.1, the latest GA release in the 7.1.x family (Data ONTAP 7.2.x isn’t supported on the F8xx series).  There are instructions for performing the upgrade from a UNIX/Linux-based system as well as instructions for upgrading the storage system from a Windows-based computer.  OK, but what about a Mac?

Well, Mac OS X is based on FreeBSD, so the UNIX instructions should work, right?  Well, not quite.  Although I mounted the NetApp’s root volume via NFS and attempted to extract the files per the instructions, it kept failing.  OK, what if I mount it via CIFS?  Same result—an error during the file extraction/file copy process and an error when running the “download” command.

Well, I can’t exactly run the Windows executable version of the Data ONTAP upgrade because I don’t run Windows.  The UNIX/Linux instructions don’t work.  Fortunately, there’s a third option—the HTTP server upgrade.  I had an OpenBSD-based web server in the lab, so within just a few minutes I had the Data ONTAP upgrade files on that server and then onto the storage system itself using the “software get” command.  The process worked quite smoothly, and I was finished upgrading the storage system a few minutes later.

I even went back and reviewed the upgrade guide again to see if it explicitly listed specific client operating systems as a prerequisite for upgrading the storage system.  The upgrade guide states that “any available version of UNIX” is required.  So why didn’t Mac OS X work?  Why did I keep getting errors running the Perl install script, yet the HTTP upgrade worked flawlessly?  Don’t get me wrong, I’m glad the upgrade worked.  I just a bit peeved that something that should have worked according to the documentation didn’t.

So, here’s a tip for all of you out there: if you ever need to upgrade a Network Appliance storage system to a newer version of Data ONTAP from a Mac OS X system, be sure to use the HTTP upgrade method.

This comparison of ZFS (Zettabyte File System) and WAFL (Write Anywhere File Layout) by Network Appliance (no direct link for WAFL) is an interesting comparison of these two advanced filesystems and their feature set.  Be sure to read the comments for some additional insight on the comparison of the two filesystems and some clarification about supported features.

One distinction raised in the comments that’s worthy to be noted here is that any comparison of this sort also, by its very nature, takes into account the operating system that runs the filesystem.  As a result, any comparison of ZFS vs. WAFL also involves, to a lesser extent, Solaris and Data ONTAP, respectively.  Similarly (and I think this may have been pointed out in the comments as well), the underlying hardware used by these filesystems (and their operating systems) also comes into play as well.

For these reasons, it’s impossible—in my mind, at least—to perform any sort of apples-to-apples comparison of these two technologies.  It’s still an interesting article, though.