Comments on: Using NetApp Deduplication with Block Storage

By: Peter Just

Peter Just — Thu, 02 Jun 2011 21:03:29 +0000

This is an old post, but a very good one. We have been running deduplication in our environment for quite a while. It seems to us that when we run a partial deduplication it appears to take longer than a “full” deduplication. This seems counterintuitive. Have you experienced this as well?

By: slowe

slowe — Thu, 23 Dec 2010 15:40:15 +0000

Mar, it’s not a stupid question—it’s actually a very pertinent question. You’re correct in that the host accessing the LUN will always only see the space allocated to the LUN. It’s on the back end that you need to monitor space usage. This makes using deduplication with block storage a task that can be quite administrator-intensive, or else you’ll run into issues like what you’ve described (and already experienced). The storage administrator has to closely monitor storage utilization on the array (even more so if you are using snapshots with your deduplicated block storage) or else you run the risk of taking LUNs (and potentially applications) offline.

Good luck!

By: Mar

Mar — Thu, 23 Dec 2010 14:36:30 +0000

Sorry for my stupidity here, but I am not a storage admin.
I have a question, so when you have deduplication enabled and you present a 100GB LUN to the server, the OS sees 100GB no matter what is going on in the background on the filer side, now say the LUN has 70GB of data, does the filer see the space allocated as 70GB or 100GB? We have recently run into issues with losing our connection to the LUNS’s and my storage administrator was seeing the space allocated on the filer smaller then what the OS was seeing.

By: Jon

Jon — Fri, 17 Sep 2010 14:34:54 +0000

Drew – I found the document at http://communities.netapp.com/docs/DOC-1192

This was located under “Documents” here:
http://communities.netapp.com/community/products_and_solutions/deduplication

Jon

By: Drew

Drew — Tue, 02 Mar 2010 15:28:04 +0000

What happened to this White Paper? The link appears to be broken now…

http://communities.netapp.com/community/our_products_and_solutions/deduplication

It sounds like the exact Doc that I’m looking for to implement DeDup on our ESX FC LUNs on NetApp…

Please let me know (anyone)

Drew

By: Larry Freeman

Larry Freeman — Wed, 21 May 2008 18:26:11 +0000

Thanks Scott. BTW I worked through 10 revisions of this document before I could get all our experts to agree on the exact behaviour of dedupe within a NetApp LUN! As we move forward with more apps running dedupe on LUNs I hope this doc becomes very useful. We are seeing broad adoption with VMware and I am starting to notice customers experimenting with dedupe on SharePoint, Oracle, Exchange and SQL. Thanks again for you great blog Scott.

By: slowe

slowe — Wed, 21 May 2008 01:55:24 +0000

Larry,

Excellent document, I just finished reading it. I may have to post an article just on that document!

By: Larry Freeman

Larry Freeman — Tue, 20 May 2008 22:54:47 +0000

Hi Scott, great information in your blogs. BTW we’ve just written a configuration guide for NetApp dedupe and LUNs. There are actually 5 basic configurations that will drive the “freed” blocks to the LUN overwrite area, the volume free pool, the aggregate free pool, or a combination of the above. I’ve posted the paper on the NetApp dedupe community site:
http://communities.netapp.com/community/our_products_and_solutions/deduplication

By: slowe

slowe — Wed, 14 May 2008 11:10:20 +0000

Gary,

I use NFS quite a bit with VMware. I haven’t had the opportunity to do any performance testing, but I can say that subjectively it appears to perform very well. Since Data ONTAP manages the file system (rather than layering a host file system on a LUN), it makes it easy to leverage Snapshots. Check out these articles for more information:

http://blog.scottlowe.org/tag/nfs

Thanks for reading!

By: Gary

Gary — Wed, 14 May 2008 08:07:09 +0000

Great article Scott. I am wrestling with VMWare and ASIS at the moment and the biggest consideration for me was volume size with regard to ASIS limits (2TB on FAS3040 I think). I am hitting performance issues at the moment with SCSI reservation errors, and it looks like I need to break out my VMs into smaller LUNs (currently 500GB LUN, thin provisioned, with 15 vms per LUN). I initially kept one LUN per volume in order to keep the management easy, but I think having several smaller LUNs should reduce SCSI reservation errors. I am hoping that moving machines to a new LUN in the same volume will keep the dedupe amount around the same (as I snapmirror the volume and don’t want to increase the transfer volume too much).

I am looking forward to Ontap 7.3 as it will dedupe at the aggregate level (which should increase my savings significantly) and they were also talking about increasing ASIS limits and SAN performance.

In the future I might move from FC to NFS, what are your experiences with this (if any)?