MetaLUNs are a way of expanding a LUN for either additional I/O capacity (using a striped MetaLUN) or additional space (using a concatenated MetaLUN). A striped MetaLUN, as the name implies, stripes data across multiple component LUNs. Each of these component LUNs resides on a different RAID group, so creating a striped MetaLUN allows the MetaLUN to utilize all the spindles in all the RAID groups. A concatenated MetaLUN, on the other hand, fills up one component LUN before moving on to the next component LUN. I/O capacity is essentially unchanged, but storage capacity is expanded.
This article has some good information on MetaLUNs.
While trying to learn more about MetaLUNs, I searched high and low for a “how to” guide on creating MetaLUNs. Surprisingly enough, I didn’t find anything. So, here’s my take on how to create a MetaLUN. EMC experts, feel free to refine my process, correct my errors, and provide general guidance.
The equipment used in my experimentation is a CLARiiON CX4-960 running FLARE 28 (as best I can tell). I performed this task using Navisphere 6 running under Internet Explorer 6 on Windows Server 2003.
OK, ready? Here we go:
- Create the necessary RAID groups to house multiple LUNs. Based on my limited experience thus far, it seems like RAID 5 (4+1) is the most common configuration for a RAID group unless you know you need something else.
- Create LUNs with the exact same size and settings in each of the RAID groups you created. For maximum performance, ensure that the LUNs are created on separate buses. I used LUNs on three separate RAID groups: one on Bus 0, one on Bus 1, and one on Bus 2.
- Right-click on the first LUN you created and select Expand.
- Click Next to start the wizard.
- Select Striping and click Next.
- Select each of the LUNs. Hold down the Control key to select more than one LUN. Click Next when you have selected all your component LUNs.
- Make sure Maximum Capacity is selected and click Next.
- Click Finish.
When the wizard is done processing, the LUNs you created in the RAID groups will be moved into the Private LUNs container, and a new MetaLUN object will appear in Navisphere (under LUN Folders > MetaLUNs). You can then present this MetaLUN out to one or more servers in the same way you would present any other LUN object.
If I’ve misrepresented something or have provided incorrect information, please let me know by speaking up in the comments. I’d also love to hear any recommendations from EMC experts on the use of MetaLUNs, advantages and disadvantages of using them, etc. Share your knowledge!
-
don’t get too carried away with striping though. some “people” think it’s great to create m-Luns out of every raid group in the array. This then means that every i/o runs a high degree of probability of colliding with another’s and thus necessitating a TON of seeking. I try to engineer storage allocation to stay within as few raid groups as possible for “silly” use (eg. operating systems) and try to dedicate (as much as possible) spindle sets to high-io tasks like SASwork, Oracle temp|redo|index. I’ve found that many storage “engineers” are downright lazy and stupid when it comes to laying out space and spindles. Maybe it has to do with vendors likewise being lazy and telling them it’s good enough to just stripe everything across every raid group.
-
This is excellent. I wish my little AX4-5s had this. Instead of this very smooth method, I’m always expanding my LUNs, then “tricking” the OS into seeing the updated size, then creating a new partition in the empty space, then expanding my filesystem over that by adding the new physical volume to the LVM system.
/sigh
-
Matt Simmons,
Careful, though–this does NOT address OS-level partitioning issues.
Matt,
Have any recommendations on this? I’d love to see any documentation around best practices.
-
There is an EMC Engineering White Paper.
EMC CLARiiON MetaLUNs Concepts, Operations, and Management
Published June 2006
EMC Part Number H1024.1The document is a quick read about 31 pages and really fills any “technical” gaps that you may have. I’ve also used EMC’s san simulator software (navi simulator) that allows you to “practice” before you do this in production.
-
How do you know that the lun’s on the 3 separate raid groups on 3 different bus’ is the best decision? Shouldn’t the OS know what’s best? Shouldn’t the OS determine it for you unless you decide otherwise? You’re still only striping the data over a handful of disks at best which is so old school. What if performance isn’t good enough? What do you do then?
@pros
-metaluns can give increased capacity or speed (how many storage groups/luns/bus’ do I use??)@cons
-scale out to 960 disks on 2 ACTIVE/PASSIVE controllers?!? You’d have to be nuts or never had a CX before. I don’t care what proc combo (dual/quad) and 64-bit OS (is it still a variant of XP? lol), scaling that far out with only 2 controllers is ridiculous even with “licensed” powerpath. Ever have a SP replaced? I have several times and it’s a hoot. Ever have the SP bug check that causes a panic/reboot and that support will swear never happens to other customers lol.If you really want some fun, get a celerra NAS and add it to your favorite flavor of CX and good luck upgrading flare code/dart code as you will be locked in to the most ass backwards support matrix.
If you do experience crappy performance even with metaluns you will eventually be blamed or snickered at for not considering the symmetrix line. (been there done that)
EMC ever hear of storage pools for the CX? Stripe data across every disk? Write to the fast track or have automated tiering/ILM? It’s almost 2010 fellas…Take a look at Compellent/3PAR/Pillar if you want to see some modern arrays that simply make sense and at a cheaper price point than a CX with far superior software. No, I don’t work for either company, I’m just an admin who had his headaches from each of the following… FC-4700′s, CX3-80′s, CX4-80′s and Celerras. I’m done with forklift upgrades and crappy inferior PS/customer support from Bangalore. Once support is up, we are ditching it all and moving to a storage company that does “storage” and isn’t interested in tons of acquisitions and how to position them…
Sorry for the negative rant
-
Disclosure – I’m an EMC employee.
Thank you Scott – the H1024 doc is the best one that I know of. You can get it on Powerlink. The key to searching Powerlink, and the Resource Library on EMC.com (http://www.emc.com/resource-library/resource-library.esp) is to put in only ONE search keyworld, then filter the results.
Yes, it’s very frustrating, and yes, we are fixing it
Note that the MetaLUN component requirement of being the same size is for a Meta restripe, and you can have any sizes for a meta concatenation operation.
Meta LUNs are handy-dandy, and along with LUN migration (moving from one underlying RAID configuration to another) are a good “get out of jail free” card if it turns out you need more (or less) IO or capacity.
LUN shrink is supported in FLARE 29, so you can literally go in just about any direction (capacity or performance) as needed.
These are the foundational technologies that will be automated more and more as FAST (fully automated storage tiering) arrives not only on the V-Max and Celerra, but also on CLARiiON.
I think more automation will help. While Metas, LUN migrations, large parity RAID groups, and thin provisioned pools are really useful, sadly most customers still use the most basic stuff only and use thick R5 4+1s all over the place (and are a lot less efficient for it).
-
I’m a CLARiiON Performance Specialist, working for EMC Technical Support, so I thought I’d chip in here, although I agree with most of the above.
Striping across every RAID Group in sight can definitely be a problem. The benchmarks for one LUN will look good, but there will be a lot of contention with other LUNs, as Matt stated above. Another issue is the management overhead. Each MetaLUN and all of its components has to be monitored and managed, which creates an overhead on the SP CPU. If you have a dozen or more component LUNs in each MetaLUN, then very soon you’ll have over a thousand objects to manage. If you then turn on Navisphere Analyzer to check on performance, that will also increase the impact as it has to track performance on each component LUN.
Another thing to watch out for is mixing striped and non-striped LUNs on the same RG, because this will lead to some parts of your MetaLUN being slower than other.
The biggest gotcha is that whatever you do, do not stripe together any LUNs which share the same RAID Group. This causes Linked contention and will drastically increase your seek distances.
The ideal for RAID 5 is generally acknowledged to be 4 * (4+1) in a RAID 5. That translates as four RAID Groups with five drives each, striped together to make a twenty drive LUN. Any other LUNs on these RG can be striped in the same way, but changing which RG is used first in each striped MetaLUN will help balance the load even more.
It’s a similar story for RAID 1/0, with the ideal configuration being 4 * (3+3); a twenty four drive MetaLUN.
The best whitepaper to refer to is titled “EMC CLARiiON Best Practices for Fibre Channel Storage” and there is a version for each Release of Firmware (the latest is Release 29). -
Trackback from uberVU - social comments on Monday, October 26, 2009 at 4:55 pm
-
Scott
I’ve never been a fan of Meta devices, although I’ve seen many implementations where wide striping has been implemented by creating the smallest possible hyper and combining those in large quantities. Some extreme examples have had a single MetaLUN spread across 128+ drives, which common sense dictates isn’t a good plan. I hope we see EMC, HDS & IBM relegate the legacy concept of metas as soon as possible. Fortunately (on the Enterprise arrays at least), wide striping is starting to do that.
Chris
-
Very interesting and enlightening discussion – thank you. I also found it odd that there seems to be little documentation on metaluns. I’ve read the H1024 document at least 5 times and still have some unanswered questions. (Maybe the answers are in there, but I don’t understand them
I am currently managing 3 CLARiiON CX3′s (two CX3-40′s and one CX3-80). About a year ago (when the arrays were fairly empty) I adopted a strategy of carving all the RAID 5 space into 100GB FLARE LUNs and binding these into MetaLUNs. I mostly followed some of the suggestions above, like trying to spread them evenly across backend busses. I also try to provision the metaluns in pairs, one owned by each SP.
One thing I missed and am backtracking to ‘fix’ is mixing component luns from different RAID group geometries (some 4+1 and some 6+1). The reason they’re not the same throughout is because on trays with hot spares there are 14 drives left so I split them into two 6+1 groups. I’m not sure what the impact of having the unmatched stripe elements is but I’m assuming that it’s not “good” and should be avoided. In the future I plan to bind a slightly different size FLARE LUN in different size RAID groups so only the correct ones show up in the “expand” dialog. I’m guessing that a difference of just 1GB or less would accomplish this.
Now I’m wondering if this approach could have some negatives, e.g. a 1TB metalun with components from 10 different RG’s engaging 50 spindles (if 4+1′s). I haven’t seen any poor performance from these that I could put my finger on but haven’t really had much time to look closely. I have Analyzer licensed so I *could* but I only look when someone complains. So far, any complaints have turned out to be “non-array” issues.
I’d be curious to know if anyone has run tests on a variety of metalun geometries to see if there are any “sweet spots” or points of diminishing returns. If my arrays weren’t pretty much full now I’d be tempted to find time to do that.Anyway, the thing that actually brought me here was trying to find out if it was actually possible to concatenate two metaluns. I can’t find a way to do it, it will only allow me to expand a metalun with more FLU’s. I’m still running FLARE 03.26.0×0.5.016 so maybe I just need newer code (which hopefully I’ll have before the end of the year).
-
Seems like an inefficient way to size LUNs? For instance – with NetApp simply create the LUN and disable space reservations. The LUN will consume zero space initially and grow automatically as data is written to it. Disclosure – I am a NetApp employee.
1. lun create -s size -t ostype lun_path
2. lun set reservation lun_path [enable | disable]Anyone tried this?
-
Like I’ve said before in numerous other venues, every architectural/technical decision has advantages and disadvantages. EMC might lose some flexibility in sizing LUNs, while NetApp might lose some flexibility in handling different types of disk workloads. Both approaches have their advantages and disadvantages. Pros and cons, my friend…
-
Great article and thread Scott.
Chad: you mentioned the new LUN shrink capability of Flare 29, any ideas if and when that will make its way into vSphere 4 VMFS volume resizing? Currently we can expand a VMFS but not in the opposite direction.
For metaLUN performance best practices, I am still unclear about the rules that govern the Flare LUN backing, ie if we have a 15x450GB 15K FC DAE with one hot swap, does the following make sense:
4 spindle RAID 1/0 + 4 spindle RAID 1/0 + 6 spindle RAID 1/0 metaLUN
Or are should we only create three 4 spindle RAID 1/0 disk groups for the metaLUN?
This is for an OLTP workload on VMFS; everything I have researched so far says that a metaLUN stripe is more efficient than a large Flare LUN stripe (14 spindle for example) based on large cache page penalties…?
-
Is it possible to grow a normal LUN as opposed to a MetaLUN?
We’re using DMX4-4500 Symmetrix here and my SAN guy is telling me we cannot grow a volume. I have a LUN attached to a SQL Server and we need an extra 50% on there. He presented me a new drive with that 50% which I could use, but its far easier if the space shows up on the existing Volume as I can go in with DiskPart on the command line and expand that volume.
-
We have some databases and the DBAs requested RAID 10 arrays. They configured 3 8disk RAID 10 groups and created a meta lun that spanned all 3 raid groups for the 6 drives. Any issues with create a RAID 10 meta lun across 3 raid 10 luns?
-
what is maximum METALUN size limit for Netapp and for EMC..???
waiting…
Site Sponsors
16 comments
Comments feed for this article
Trackback link: http://blog.scottlowe.org/2009/10/23/how-to-create-a-metalun/trackback/