3010: A MultiStore Primer

This session describes NetApp’s MultiStore functionality. MultiStore is the name given to NetApp’s functionality for secure logical partitioning of network and storage resources. The presenters for the session are Roger Weeks, TME with NetApp, and Scott Gelb with Insight Investments.

When using MultiStore, the basic building block is the vFiler. A vFiler is a logical construct within Data ONTAP that contains a lightweight instance of the Data ONTAP multi-protocol server. vFilers provide the ability to securely partition both storage resources and network resources. Storage resources are partitioned at either the FlexVol or Qtree level; it’s recommended to use FlexVols instead of Qtrees. (The presenters did not provide any further information beyond that recommendation. Do any readers have more information?) On the network side, the resources that can be logically partitioned are IP addresses, VLANs, VIFs, and IPspaces (logical routing tables).

Some reasons to use vFilers would include storage consolidation, seamless data migration, simple disaster recovery, or better workload management. MultiStore integrates with SnapMirror to provide some of the functionality needed for some of these use cases.

MultiStore uses vFiler0 to denote the physical hardware, and vFiler0 “owns” all the physical storage resources. You can create up to 64 vFiler instances, and active/active clustered configurations can support up to 130 vFiler instances (128 vFilers plus 2 vFiler0 instances) during a takeover scenario.

Each vFiler stores its configuration in a separate FlexVol (it’s own root vol, if you will). All the major protocols are supported within a vFiler context: NFS, CIFS, iSCSI, HTTP, and NDMP. Fibre Channel is not supported; you can only use Fibre Channel with vFiler0. This is due to the lack of NPIV support within Data ONTAP 7. (It’s theoretically possible, then, that if/when NetApp adds NPIV support to Data ONTAP that Fibre Channel would be supported within vFiler instances.)

Although it is possible to move resources between vFiler0 and a separate vFiler instance, doing so may impact client connections.

Managing vFilers appears to be the current weak spot; you can manage vFiler instances using the Data ONTAP CLI, but vFiler instances don’t have an interactive shell. Therefore, you have to direct commands to vFiler instances via SSH or RSH or using the vFiler context in vFiler0. You access the vFiler context by prepending the “vfiler” keyword to the commands at the CLI in vFiler0. Operations Manager 3.7 and Provisioning Manager can manage vFiler instances; FilerView can start, stop, or delete individual vFiler instances but cannot direct commands to an individual vFiler. If you need to manage CIFS on a vFiler instance, you can use the Computer Management MMC console to connect remotely to that vFiler instance to manage shares and share permissions, just as you can with vFiler0 (assuming CIFS is running within the vFiler, of course).

IPspaces are a logical routing construct that allow each vFiler to have its own routing table. For example, you may have a DMZ vFiler and an internal vFiler, each with their own, separate routing table. Up to 101 IPspaces are supported per controller. You can’t delete the default IPspace, as it’s the routing table for vFiler0. It is recommended to use VLANs and/or VIFs with IPspaces as a best practice.

One of the real advantages of using MultiStore and vFilers is the data migration and disaster recovery functionality that it enables when used in conjunction with SnapMirror. There are two sides to this:

  • “vfiler migrate” allows you to move an entire vFiler instance, including all data and configuration, from one physical storage system to another physical storage system. You can keep the same IP address or change the IP address. All other network identification remains the same: NetBIOS name, host name, etc., so the vFiler should look exactly the same across the network after the migration as it did before the migration.
  • “vfiler dr” is similar to “vfiler migrate” but uses SnapMirror to keep the source and target vFiler instances in sync with each other.

It makes sense, but you can’t use “vfiler dr” or “vfiler migrate” on vFiler0 (the physical storage system). My own personal thought regarding “vfiler dr”: what would this look like in a VMware environment using NFS? There could be some interesting possibilities there.

With regard to security, a Matasano security audit was performed and the results showed that there were no vulnerabilities that would allow “data leakage” between vFiler instances. This means that it’s OK to run a DMZ vFiler and an internal vFiler on the same physical system; the separation is strong enough.

Other points of interest:

  • Each vFiler adds about 400K of system memory, so keep that in mind when creating additional vFiler instances.
  • You can’t put more load on a MultiStore-enabled system than a non-MultiStore-enabled system. The ability to create logical vFilers doesn’t mean the physical storage system can suddenly handle more IOPS or more capacity.
  • You can use FlexShare on a MultiStore-enabled system to adjust priorities for the FlexVols assigned to various vFiler instances.
  • As of Data ONTAP 7.2, SnapMirror relationships created in a vFiler context are preserved during a “vfiler migrate” or “vfiler dr” operation.
  • More enhancements are planned for Data ONTAP 7.3, including deduplication support, SnapDrive 5.0 or higher support for iSCSI with vFiler instances, SnapVault additions, and SnapLock support.

Some of the potential use cases for MultiStore include file services consolidation (allows you to preserve file server identification onto separate vFiler instances), data migration, and disaster recovery. You might also use MultiStore if you needed support for multiple Active Directory domains with CIFS.

UPDATE: Apparently, my recollection of the presenters’ information was incorrect, and FTP is not a protocol supported with vFilers. I’ve updated the article accordingly.

Tags: , , , , , , , , , , , ,


  1. TimC’s avatar

    A couple of points:
    The biggest thing you missed, and the primary use case I typically see for vfilers, is that each vfiler can join a different active directory domain. Meaning you could have one physical filer, and have it be your primary fileserver for 5 different domains in a company.

    I’m not sure what you mean by “vfilers’s don’t have an interactive shell”. When you type “vfiler context vfilerX”, it puts you at the prompt of vfilerX, in an interactive shell.

  2. slowe’s avatar

    Actually, TimC, I did point out the multiple AD domain use case; see near the end of the article.

    The point about interactive shells is that, to my understanding, you cannot telnet or SSH to a specific vFiler. Correct?

  3. TimC’s avatar

    Ahh, ADD tends to kick in by the end of stories, my bad :)

    Yes, you are correct in that you can’t go directly at it. That would be works as designed :D

    I would imagine they’re leveraging some iteration of BSD jails underneath the covers.

  4. Ausmith1’s avatar

    You can’t directly ssh to a specific vfiler context but if you you use a scriptable ssh tool (such as SecureCRT) you can have it automatically log into the desired context, see example below:

    usbvtrisicr200> vfiler context usbvtcommonfiles
    usbvtcommonfiles@usbvtrisicr200> Wed Apr 8 22:18:24 EDT [[email protected]:notice]: Console context was switched to a vFiler(tm) unit usbvtcommonfiles.


  5. William’s avatar

    Can the vFiler be managed by NetApp OnTap SDK? I recently came across this blog awhile back and it had mentioned Powershell and NetApp: http://get-admin.com/blog/?p=545

  6. JeffA’s avatar

    I attended this session at Insight last year and presented a very similar session to the Charlotte Tech OnTap Live group last fall.

    One small thing to add is that you can use qtrees instead of volumes for the vfiler root. That was a big deal back in the Data ONTAP 6.X days.

    Good stuff as usual, Scott.

  7. TimC’s avatar

    @JeffA: That it was. But with flexvol’s, using qtree’s is significantly less interesting. Qtree’s in general I would argue have basically become a moot point in 99% of the environments out there with the advent of flexvol’s.

  8. Fletch’s avatar

    Scott, great write up – we’ve been running our VI off Netapp NFS vFilers for close to 2 years now. We turned on dedup with the 7.3 upgrade – was great to see the instant (well overnight anyway) savings 5Tb!
    We use vfiler dr between 2 campus area clusters and also to a 3rd dr site (chained snapmirrors) for our website content.
    I can’t wait until Netapp updates their SRA to support VMWare SRM to automate DR…as it stands the vfiler dr failover and back steps are fairly quick…

  9. JeffA’s avatar

    I totally disagree Tim. Qtrees are useless in only 98% of environments at most!

    Actually, one real benefit to Qtrees is SnapMirror. Back when I used to deploy systems instead of just sell them, I would make the root of the flexvol empty and put in a single Qtree. Then I would build Qtree SnapMirror (QSM) relationships instead of Volume SnapMirrors (VSM). The benefit is that the volume sizes could be different and I could resize the volumes more easily than with VSM. Sure, there is a performance penalty, but in most of the environments I worked in the perf penalty was never felt and the easy flexibility was good to have in your pocket.

  10. Dan C’s avatar

    William> Sure, it can.

  11. Vaughn’s avatar

    I expanded this discussion by discussing intergrating vFilers with VMware SRM. Check it out if interested.



  12. Scott Gelb’s avatar


    Cool you posted this…thank you. I only caught it a month late :) Roger posted our presentation to the partner portal on the NetApp Insight page, and he also posted a lighter version of it on PartnerCenter for employees and partners to reference. Also, I can send you or ftp the 11 minute avi video from the presentation. Nothing proprietary and running from my laptop showing both vfiler migrate and vfiler dr.

    MultiStore does support FTP. Status of a vFiler below that shows all protocols for a vFiler. All are allowed by default (we have a request in for future ONTAP to disallow by default then have the administrator allow protocols they want).

    Allowed: proto=rsh
    Allowed: proto=ssh
    Allowed: proto=nfs
    Allowed: proto=cifs
    Allowed: proto=iscsi
    Allowed: proto=ftp
    Allowed: proto=http

    One other note… vfiler dr doesn’t change the volume option “fs_size_fixed” from on to off with “vfiler dr activate”. So, if the DR site is running for a long period of time before resync back to the repaired source, and you need to grow a volume, you’ll have to turn this option off to utilize additional space added to a volume.

    To elaborate on qtrees, we used to use qtrees more with vFilers before ONTAP 7G (to better utilize legacy traditional volumes). With FlexVols we have the same utilization benefit we had with qtrees on tradvols. When using FlexVols instead of qtrees, vFiler dr/migrate use volume mirror which give the advantage of keeping dedup over the wire (not rehydrating data like qsm), and also if we leverage snapmover (vfiler migrate -m nocopy) then we need the vFiler to own all FlexVols in the aggregates used for the vFiler. Also, if an nfs mount is on a qtree and the parent volume of the qtree is not owned by the vFiler, a vFiler migrate will migrate the qtree with qsm and the target qtree will not be accessible without a client remount (volume fsid isn’t changed). However, if the parent FlexVol is owned by the vFiler, then the qtree mount is accessible since VSM is used and the fsid and file handles are moved to the target FlexVol. Nothing against qtrees though…I still use them a lot but don’t assign individual qtrees to vFilers.

    We do some cool things with vFilers for DMZ readonly data sharing…use a FlexClone or a SnapMirror of data in one vFiler to another. You can use the loopback adapter but running the mirror from vfiler0, then you don’t need any network connection between the vfilers and it’s still secure but can have a copy of any data you want…

    For management, definitely a lot of cli. If customers use the cli already they are comfortable. if they use FilerView, then they have some learning curve. API has support for vFilers…so hopefully the new NSM will support vfilers in a future release.

    Another note…it’s worth a day (often less than a day) of PS to get MultiStore setup properly….it’s not rocket science, but those who work with it a lot can do in minutes what might take others hours so it saves cost to pay for a day or half a day for setup. For example, the /etc/rc file isn’t always updated correctly, but is easy to edit after vfiler creation/configuration. For example, if you use an ip alias or vlan, sometimes the commands are put above the vif creation. For example, the alias to vif10 is made before vif10 is created, so on boot the alias will fail. Also, it is important to add the route for the vfiler if needed “vfiler run route add default” in the rc file. Also, a lot of customers implement MultiStore after the system is already running…so we move the resources in vFiler0 to another vFiler that can be used for DR…in this case we often don’t want to initialize mirrors (vfiler dr configure will initialize mirrors even if initialized), but instead we just manually mirror the vfiler root volume created on the source, then create the dr vfiler manually, then resync it…saves the pain of re-init of a mirror.

    Another cool thing… ONTAP 7.3.1 added one of the items on our wish list for the secure dr feature using ssl, so we don’t have to have rsh enabled for the controller to controller communication. “vfiler dr configure [-c secure]” 7.3.1 also put sis (dedup) commands in the vFiler context now too, so it’s configurable from within the vFiler itself now.

    MultiStore Rocks :)

    -Scott Gelb

  13. slowe’s avatar

    Scott G,

    Don’t worry about catching it late–after all, I posted it many, many months late. :-)

  14. Dmitry’s avatar

    About security. If every vFiler manages its VLANs, this situation creates security hole in case of sharing a few physical Ethernet interfaces with a few vFilers. How NetApp handle it?

    On the other hand, if a vfiler manages dedicated physical interface(s) in the exclusive usage where to get 64 + 1 Ethernet interfaces?

  15. slowe’s avatar

    Dmitry, that’s a great question. I don’t have the answer—perhaps another reader can shed some light on that question?

  16. Dmitry’s avatar

    >A key point though is that the vFiler admin cannot create VLANs

  17. satya’s avatar

    I am novice, will be posting soon

  18. Tenaro’s avatar

    I found a definition for vfiler migration where it says that is not moving data but vfiler configuration only. And you are saying that “vfiler migrate allows you to move an entire vFiler instance, including all data and configuration”. Seems to me that definition I found makes more sense otherwise there is no difference between vfiler migration and data motion. Can you please comment this? I also must add I’m new in netapp world and it is possible I figured it all wrong.

  19. Darren’s avatar

    Hey guys,

    I am looking for a reference setup for managing backups from a DMZ vFiler. Ideally I would like to be able to pull SnapVaults from the DMZ vFiler (say, vFiler1) via the interfaces on vFiler0 so that we don’t have to push our vault traffic through a firewall.

    I’m about to kick this around in a lab, but it would be nice to start with a known good sample, as well as the background to know that is possible. Worst case I can point to the DMZ vFiler’s interfaces to reference the volumes it owns, but if I could possibly pull the data via vFiler 0 that would be the best case scenario…

Comments are now closed.