iSCSI

NetApp recently published a white paper summarizing some tests they ran to compare storage protocol performance in a VMware Infrastructure environment. The white paper, TR-3697, compares the storage performance of Fibre Channel, software iSCSI, and NFS against a couple of different NetApp storage systems.

I won’t go into all the sordid details here—you can read the white paper yourself—but the end results look something like this:

• Fibre Channel provided the highest throughput and the lowest processor utilization of all the storage protocols.
• Software iSCSI provided only slightly lower throughput than Fibre Channel (not more than 9% or 10% less than Fibre Channel depending upon the specific tests being run). However, software iSCSI consistently showed the highest CPU utilization on the ESX hosts.
• NFS showed throughput on the same levels as software iSCSI (again, not more than about 9% or 10% less than Fibre Channel depending upon the tests being run) and had higher CPU utilization than Fibre Channel. However, the CPU utilization was lower than with software iSCSI.

While overall performance was roughly comparable between all three storage protocols, depending upon the tests being run, the host CPU utilization was a different story entirely. In some cases, software iSCSI’s CPU utilization was as much as 80%—that’s right, almost double—that of Fibre Channel. In no cases did the CPU utilization drop below 40% higher than Fibre Channel. Keep in mind these numbers are relative to Fibre Channel. So if Fibre Channel used 200MHz of host CPU power and software iSCSI used 360MHz of host CPU power, that’s an 80% relative increase. We don’t know, unfortunately, how this translates into actual host CPU usage; in my mind, that’s a key piece of information that really should have been included. I’m puzzled as to why it’s not included.

NFS fared better; at its worst, the tests showed NFS running CPU overhead 40% greater than Fibre Channel. At its best, NFS looked like it was only requiring about 15% more CPU overhead than Fibre Channel (keep in mind the comments made above regarding relative utilization). Of course, NetApp loves to push the NFS; the document adds the extra sell for NFS:

While NFS does not quite achieve the performance of FC and has a slightly higher CPU utilization, it does have some advantages over FC that should be considered when deciding which protocol to deploy. Running on a standard TCP/IP network, NFS does not require the expensive Fibre Channel switches, host bus adapters, and Fibre Channel cabling that FC requires, making NFS a lower cost alternative of the two protocols. Additionally, operational costs are low with no specialized staffing or training needed in order to maintain the environment. Also, NFS provides further storage efficiencies by allowing on-demand resizing of data stores and increasing storage saving efficiencies gained when using deduplication. Both of these advantages provide additional operational savings as a result of this storage simplification.

I suppose I can’t blame them; NFS is one of their strong points, so they’ll naturally lean that direction.

There are a few key things that I need to say about this document, though:

1. Benchmark tests can be made to say just about anything. It’s all in the types of tests that you run and the parameters of those tests. I’m not saying that NetApp specifically skewed the tests in any way; what I am saying, though, is that users need to take these types of benchmark tests as a general guideline and not the definitive word.
2. While NetApp does highlight the “operational savings” of NFS, what they fail to mention is the added complexity of scaling NFS traffic as the environment grows. Fibre Channel multipathing in a VMware environment is very robust, and I expect that the Round Robin pathing policy will move from “experimentally supported” to fully supported rather quickly. This makes it quite easy to scale the FC connection, although to be honest that probably won’t be necessary. However, to scale the NFS connection, you need multiple NFS exports with multiple IP addresses, link aggregation via LACP/802.3ad/EtherChannel and switches that support cross-switch link aggregation, and possibly multiple VMkernel ports on different IP subnets. This is described, by the way, in the latest revision of TR-3428, also from NetApp. (As a side note, I believe that these scaling issues would affect any NFS storage vendor and are not specific to NetApp in any way.)
3. If you look at VMware’s development, you will see that Fibre Channel gets the goods the earliest. iSCSI and NFS were only added in VMware Infrastructure 3, whereas Fibre Channel support has been around in ESX for much longer. Storage VMotion support went to Fibre Channel first. VCB support went to Fibre Channel first. SRM support went to both iSCSI and Fibre Channel, but not NFS. Fibre Channel multipathing is, as I mentioned already, quite robust; iSCSI multipathing and NFS multipathing aren’t quite so robust. All these things considered, there could be a sound business case to use Fibre Channel in spite of cost savings from iSCSI (especially software iSCSI, given the added CPU overhead) or NFS. That’s something that each individual organization will need to decide for themselves.

By the way, I know the gentleman that wrote this technical report and he’s a straight-up guy. I respect him. So, don’t take any of my comments or thoughts to imply anything beyond the fact that I’m simply presenting my thoughts around the data contained in this document. You should also know that I am a fan of using NFS for VMware, but I don’t necessarily believe that it is the “slam dunk” that it’s often presented to be.

UPDATE: I’ve made some corrections to the interpretations of the CPU utilization numbers in response to some of the comments below.

Here’s Virtualization Short Take #12, a collection of links I’ve gathered over the last week or so and my thoughts on them. Enjoy!

• For those that missed it in the Release Notes, VMware added support for Storage VMotion and 10Gb Ethernet with iSCSI SANs, as outlined in this VI Team blog entry. I went back and reviewed the Release Notes and didn’t see this listed anywhere, so this is news to me. Of course, I already knew that Storage VMotion worked just fine with iSCSI, but this added formal support for iSCSI.
• Virtualfuture.info published some good recommendations for running Citrix in a VI3 environment. If you run Citrix Presentation Server…er, XenApp…in a VI3 environment, these tuning tips may prove quite handy.
• VMware’s Virtual Reality blog posted an entry on some of the architectural advantages of VMware Infrastructure in comparison to the two leading competitors, Xen (any Xen-based solution) and Hyper-V. Many of the things listed as advantages by VMware are severe points of contention with the other vendors, such as the direct vs. indirect I/O model. Ultimately, time will tell which model was the best; I honestly don’t know enough about the deep dark internals to really state which is better. One thing I am glad to see pointed out is the true comparison of hypervisor sizes; Microsoft can say all they want that Hyper-V is only 600K in size and therefore is the “thinnest” hypervisor, but the truth of the matter is that Hyper-V can’t run without Windows Server 2008 in the parent partition. As a result, it doesn’t really matter how “thin” Hyper-V is, does it?
• Via Mike Laverick, I learned that Microsoft may have brought up the whole 64-bit hypervisor vs. 32-bit hypervisor argument yet again. Mike used a snippet from this Microsoft Virtualization Team Blog entry; in reading it myself, I don’t get quite the same 64-bit vs. 32-bit that Mike picked up. That’s good, because I didn’t want to have to go there again. Personally, the tone I picked up from the whole article was one of educating people far too accustomed to Virtual Server/VirtualPC and trying to educate them on how Hyper-V is different.
• Virtualization analyst Chris Wolf recently posted an entry in which he questioned if Apple would capitalize on the opportunity that virtualization is creating. It’s an interesting scenario, one that is similar to a scenario that I discussed a couple of years ago in a piece titled “Application Agnosticism.” In that article, I suggested that seamless host-guest interactions with virtualization software (now implemented by VMware as Unity and by Parallels as Coherence) would usher in a new wave of computing. I suggested that Mac OS X was ahead of the curve because of its ability to run native OS X applications, UNIX applications, X11 applications, Windows applications via WINE (or the commercial variant CrossOver Office), and applications from any other operating system via virtualization. Sounds like I may have been a bit ahead of my time!
• Chad continues discussing VMware HA with another post on some additional configuration options for HA. Also check out the comments with links to even more information on HA’s advanced configuration options.
• This VMware KB article has some good information on getting LUN identification information. The breakdown of the command-line output from esxcfg-mpath is particularly helpful (and for that reason I’ve added it to my del.icio.us bookmarks).
• Rich of VM /ETC shares with us a “Doh!” moment he had when he saw this simple method for identifying VMs with snapshots. Sometimes it’s the simplest solutions that evade us the longest. Here’s what I want to know: Aaron, what exactly does “/HEADDESK” mean, anyway?
• This article at SearchNetworking.com brings to light some of the challenges networking professionals face with server virtualization. I do agree with one point made in the article regarding the mapping of applications—what the end users really care about—to the networking infrastructure. VMware’s support for CDP in recent versions of VMware Infrastructure is a step in the right direction, but there is still more work to do for sure. I’m not so sure about the rest of the points in the article, but I may be an exception to the norm; I was a CCNA for a while (on track for CCNP) and have done my fair share of Cisco configurations, so I’m no stranger to the networking world. The use of VLANs to ease configuration in a server virtualization environment seems just second nature to me. Also, I did note that the author indicated that “server administrators sometimes inappropriately configure the switches to create a loop” (referring to vSwitches in ESX). How exactly does that happen? I’ve never seen a way to link two vSwitches together without using a VM.

As always, readers’ thoughts are welcome in the comments!

Like everyone else in the virtualization world (except for perhaps the folks in Palo Alto, CA), there’s a lot of Hyper-V stuff crossing in front of me.

This time it’s an article on storage options for Hyper-V, written by Jose Barreto. (You’ll recall that I referenced Jose’s clustering article a few days ago.) Out of the wide variety of blogs coming out of Microsoft, Jose’s is one that I have really, truly found informative and helpful. The home page for his blog is here.

Jose also wrote a follow-up article on Hyper-V’s storage options where he discussed booting from iSCSI.

Great work, Jose! Keep it coming.

What I had hoped to be able to publish today would be an article describing how to configure and use ESX’s software iSCSI initiator as a failover path for Fibre Channel, so that if the Fibre Channel fabric completely failed VM traffic would automatically failover to software iSCSI. I thought that this would be a great, low-cost way to add another layer of redundancy to your VMware ESX environment.

Unfortunately, I can’t make it work. Here’s the setup I’ve been using for testing:

• A 200GB LUN visible to ESX over both Fibre Channel (FC) and software iSCSI
• A VM, stored on this LUN, running Windows Server 2003 R2

Initial tests led me to believe that it would indeed work. I verified that both the FC path as well as the iSCSI path were listed as separate paths for the same LUN. Without placing any load on the VM, I pulled the FC connection from the back of the server. The VM stayed up, and I was able to browse the local hard drive inside the VM. Network connectivity remained active. And the “Manage Paths” dialog box even showed the FC connection as “Dead” and the iSCSI connection as On/Active. Given that information, it seemed like all was good.

Determined to verify that it was working as I expected, I trotted out a copy of IOmeter and tried to repeat the tests. This time around, though, the tests did not go quite so well. IOmeter showed that disk throughput stopped, and the VI Client locked up. I repeated this set of tests a couple of times, and each time—while IOmeter was running—I ran into issues.

Based on these results, I’m inclined to say that one of two things is true. Either:

1. I did something very, very wrong; or
2. ESX isn’t quite right to support automatic failover between FC and software iSCSI.

Has anyone else tried this, or am I the only one? If you have tried it, did it work? If so, what steps did you have to take—if any—to make it work properly?

Building on my earlier article on setting up NetApp deduplication, I wanted to follow up with some information on using NetApp deduplication with block storage (LUNs presented via Fibre Channel or iSCSI).

For the most part, using NetApp deduplication with block storage is a lot like I described earlier:

• You (obviously) still need the NearStore and deduplication (A-SIS) licenses installed on the controller(s).
• You will still turn deduplication on using the “sis on” command for the FlexVol containing the LUNs.
• Limitations on the size of the FlexVol still apply.
• You use the “sis status” command to check on the status of deduplication, and the “sis config” command to see the deduplication schedule.

OK, so what’s different? Well, it has to do with how LUNs are provisioned on a NetApp storage system. I’ve blogged before about managing LUN space requirements on a NetApp, and about using LUN clones vs. FlexClones. That second article, in particular, really goes into detail on how LUNs are implemented on top of NetApp’s file system, WAFL. Since LUNs are represented by WAFL as a single file, they are also normally “space reserved,” meaning that the maximum size of the LUN is allocated at the time of creation. If you create a 50GB LUN, then Data ONTAP creates a 50GB file right away. (For readers out there who are well-versed in NetApp storage, I know that’s a bit of a simplification, but bear with me.)

What does this have to do with deduplication? Great question. If the LUN is space reserved—meaning that the maximum size of the LUN is allocated up front and remains allocated to the LUN—then the file that represents the LUN won’t ever decrease in size to reflect deduplication savings, and deduplication therefore does you absolutely no good whatsoever. This is not to say that deduplication doesn’t work, just that it won’t help you at all.

Fortunately, there’s an easy fix for this. When creating the LUN, simply uncheck the box marked “Space Reserved” and allow Data ONTAP to allocate space to the LUN out of the containing FlexVol on an as-needed basis. Because the file that represents the LUN can grow in size, it can also shrink in size, and deduplication will cause the file that represents the LUN to decrease in size. This then allows you to provision additional LUNs from the same FlexVol to take advantage of the space savings resulting from deduplication.

I know that seems a bit confusing; I’ll probably post another article with some more in-depth discussions of the details. (Either that, or I’ll encourage my NetApp readers to chime in below in the comments.)

So, in summary, when using NetApp deduplication with block storage:

• you’ll setup and configure deduplication on the FlexVol containing your LUN(s) just like described in my earlier article;
• you’ll uncheck the “Space Reserved” checkbox when creating the LUNs to be deduplicated;
• you won’t see the space savings from the host’s perspective and therefore can’t store more data in that LUN than the size of the LUN; but
• you will be able to provision additional LUNs in that same FlexVol that can be presented back to host for additional storage.

I hope this helps clarify some of the questions or issues surrounding the use of NetApp deduplication with block storage. Feel free to add information, experiences with deduplication and block storage, or ask additional questions in the comments below.

UPDATE: There are some additional considerations about how to provision LUNs along with NetApp deduplication that warrant a more in-depth discussion. Look for a follow-up post within the next few days.

With the release of VMware Infrastructure version 3.5, VMware added support for jumbo frames. Although the documentation states that jumbo frames “are not supported for NAS and iSCSI traffic”, jumbo frames for NFS and iSCSI does actually work. Here’s some information on getting it working.

How I Tested

Keep in mind that this is not an “officially supported” configuration (see the section on the “Official” Support Statement below), so use at your own risk. I will not be held responsible if you blow up your production environment trying to make jumbo frames work.

Here’s how I tested the use of jumbo frames for software iSCSI and NFS datastores:

• For the physical switch infrastructure, I used a Cisco Catalyst 3560G running Cisco IOS version 12.2(25)SEB4.
• For the physical server hardware, I used a group of HP ProLiant DL385 G2 servers with dual-core AMD Opteron processors and a quad-port PCIe Intel Gigabit Ethernet NIC.
• For the storage system, I used a NetApp FAS940 running Data ONTAP 7.2.4.

The exact commands and/or procedures may be different for you depending upon the hardware and/or software versions that you’re running in your environment. Keep that in mind.

Configuring the Physical Switch

Fortunately for me, the Cisco Catalyst 3560G does indeed support jumbo frames. (Naturally, you’ll want to ensure that your switch supports jumbo frames.) Jumbo frames are not, however, enabled by default; they must be enabled using the following command in global configuration mode:

system mtu jumbo 9000

Note that 9000 bytes seems to be the generally accepted size for jumbo frames, so that’s what I used.

After running this command, you must reboot the switch. The change doesn’t take effect until a reload. Fortunately, IOS reminds you of this after you enter the command. Once the switch has rebooted, you can verify the MTU setting with this command:

show system mtu

This should report that the system jumbo MTU size is 9000 bytes, confirming that the switch is ready for jumbo frames. Now we’re prepared to configure the storage system.

Configuring the Storage System

Using FilerView, increasing the MTU on the appropriate network interfaces to 9000 bytes is as simple as going to Network > Manage Interfaces and then clicking the Modify link for the interface to be changed. Set the “MTU size” to 9000 (from the default of 1500), click Apply, and you’re ready to roll.

You can verify the settings in FilerView using Network > Manage Interfaces > Show All Interface Details, or by using the “ifconfig -a” command from a Data ONTAP command prompt.

Configuring ESX Server

There is no GUI in VirtualCenter for configuring jumbo frames; all of the configuration must be done from a command line on the ESX server itself. There are two basic steps:

1. Configure the MTU on the vSwitch.
2. Create a VMkernel interface with the correct MTU.

First, we need to set the MTU for the vSwitch. This is pretty easily accomplished using esxcfg-vswitch:

esxcfg-vswitch -m 9000 vSwitch1

A quick run of “esxcfg-vswitch -l” (that’s a lowercase L) will show the vSwitch’s MTU is now 9000; in addition, “esxcfg-nics -l” (again, a lowercase L) will show the MTU for the NICs linked to that vSwitch are now set to 9000 as well.

Second, we need to create a VMkernel interface. This step is a bit more complicated, because we need to have a port group in place already, and that port group needs to be on the vSwitch whose MTU we set previously:

esxcfg-vmknic -a -i 172.16.1.1 -n 255.255.0.0 -m 9000 IPStorage

This creates a port group called IPStorage on vSwitch1—the vSwitch whose MTU was previously set to 9000—and then creates a VMkernel port with an MTU of 9000 on that port group. Be sure to use an IP address that is appropriate for your network when creating the VMkernel interface.

To test that everything is working so far, use the vmkping command:

vmkping -s 9000 172.16.1.200

Clearly, you’ll want to substitute the IP address of your storage system in that command.

That’s it! From here you should be able to easily add an NFS datastore or connect to an iSCSI LUN using jumbo frames from the ESX server.

“Official” Support Statement

Officially, jumbo frames are only supported by VMware for use by virtual machines. Technically, VMware does not support the use of jumbo frames for the software iSCSI initiator or for use with NFS datastores. At least, that’s my understanding.

So, feel free to tinker around with jumbo frames for IP-based storage, and when VMware adds official support for it in the future—I can’t imagine why they wouldn’t—then you’ll be able to hit the ground running with the configuration steps necessary to make it work.

As sometimes happens, I’ve been collecting a variety of virtualization-related links that, while interesting, don’t necessarily warrant a full blog posting by themselves.  Since I don’t want to just copy someone else’s content and not provide any value of my own, I’ve decided to start irregularly publishing a “Virtualization Short Take”.  Each of these will just be a few links with a brief commentary or thought attached.  Perhaps these will lead to broader and more active discussions around some of these topics.

Without further delay, then, here’s the first-ever Virtualization Short Take:

• Via Duncan, a new and undocumented feature in VCB’s config.js file that allows for non-quiesced snapshots of virtual machines.  I’d be interested in more discussion around why some workloads might be better served by non-quiesced snapshots, so if anyone has some insight please speak up.
• Also via Duncan, fixes for crashes caused by installing the Converter plugin to VirtualCenter.  Two solutions are available; try this one, then try this approach.
• Via Thomas, it looks like VMware has published a storage performance white paper comparing Fibre Channel, hardware iSCSI, software iSCSI, and NFS.  I just finished reviewing the document myself, and plan to go back and look at it again more thoroughly.  It’s useful information for virtualization architects or engineers responsible for designing storage solutions for VMware.
• In case anyone’s interested in creating their own bootable ESX 3i USB drive, here’s more information.  This is something I need to take a closer look at myself, so when I have the chance to run through these instructions I’ll try to post some feedback on how well they worked.
• Based on some interaction with a customer yesterday, it looks as if VirtualCenter 2.5 may require the Power User role to be directly assigned to the Hosts & Clusters object in order for users to be able to attach local (client-side) ISO files to a VM.  Has anyone else seen this behavior?  I’m going to try to recreate the problem myself, but if you’ve seen this please speak up in the comments.
• I’ve also recently received word from a friend that the thin-provisioned disks VMware uses by default on an NFS datastore may not be honored when cloning VMs from a template.  Again, I plan to test this myself, but if anyone else out there has seen this behavior I’d love to hear about it.

As always, I’d love to hear your thoughts, so feel free to add your voice in the comments below.

I wanted to take just a brief moment and let everyone know about a few articles that I’ve got “in the pipeline” for the site.  If there is one—or more—of these articles that looks particularly interesting, speak up in the comments.

Here are the articles that are currently under development:

• An update to the Solaris-AD integration instructions:  Last time I ran through these instructions I came across a number of discrepancies and little “gotchas.”  I need to incorporate the workarounds into the integration instructions and publish a new version.
• A brief blurb on NetApp OSSV:  NetApp’s Open Systems SnapVault (OSSV) is a pretty cool technology, so I want to take a quick look at setting it up.  I’m also exploring to see what kind of unique synergies may arise from using OSSV in a VMware environment.
• Restricting access to ESX Server when using AD integration:  As reader Scott Garrett points out in this comment to an earlier article, ESX Server’s version of sshd doesn’t support the UsePAM directive.  This prevents us from using group membership in Active Directory to control access to ESX Server’s Service Console.  Or does it?  I have a hunch that there may be at least one workaround for this problem; once I’ve tested it, I’ll document it here.
• New iSCSI functionality in ESX Server 3.5:  VMware appears to have refreshed the software iSCSI initiator in ESX Server 3.5, so I want to take a closer look at and discuss here some of this new functionality.

That’s it for now, although I’m certainly open to other items that pique reader interest.  Feel free to submit your suggestions in the comments.  Thanks!

I guess I’m on a bit of a NetApp kick this week.  After discussing (or perhaps revisiting) the idea of recovering files inside VMs using NetApp Snapshots (first here late last year, then again here), I wanted to take a closer look at full VM recovery using NetApp Snapshots.

First of all, it should go without saying that you should never use any of the procedures I’m describing here without first testing them yourself.  While they worked fine for me, they may not work fine for you.  Don’t just assume they will!  Do the due diligence and test it in your environment first; you’ll be glad you did.

Second, before using NetApp Snapshots to recover VM data (file-level or full VM), be sure you are getting good, consistent Snapshots.  The Network Appliance Technical Reports Library has a number of excellent articles on this subject; I’ll defer you there for more information.

I’ll break this article into two sections, one for block-level storage (I’m using iSCSI, but the process should be almost identical for Fibre Channel) and one for NAS/NFS.  Please note that I’m not focusing so much on the specific steps that are required as I am on general concepts and any gotchas that may arise during the process.

Full VM Recovery using Block Storage

To recover a full VM using block-level storage, a number of steps have to be taken:

1. Create a LUN clone (or a FlexClone) of the original LUN based on a Snapshot. 
2. Enable resignaturing on the ESX host(s) that will need to see the cloned LUN.
3. Mount the cloned LUN(s) on the ESX host(s) and copy the appropriate VM files from the clone to the production LUN.

For the first two steps, I’ll refer you back to one of my first articles on VMware data recovery with Snapshots, which has more information on the necessary commands and settings.

For the third step, you’ll need to login to the Service Console (typically via SSH) and copy the desired VM(s)—and all their files—from the cloned datastore to the production datastore, overwriting whatever is in the destination (you typically wouldn’t need to recover a full VM unless the production VM was hosed, right?).  Once the file(s) have been copied back over to the production datastore, dismount the cloned datastore and destroy it.

You should now be able to boot up your VM at the state it was in at the time of the Snapshot used to recover it.  Unless the Snapshot was a cold Snapshot (taken while the VM was powered off), the VM will perform a file system check (chkdsk or fsck) when it boots up.

Full VM Recovery using NFS

The procedure for recovering full VMs when using NFS is even easier:

1. Using an NFS client, mount the NFS export and navigate to the hidden “.snapshot” directory.
2. In the “.snapshot” directory, find the Snapshot from which you wish to recover the VM.
3. Copy that VM’s files (the entire folder) out of the “.snapshot” directory into the production filesystem, replacing the current contents (again, this assumes that what’s in the production filesystem is no good, else why would you be recovering a full VM?).
4. Unmount the NFS export from your NFS client.

The recovered VM should now boot and be back to the point in time at which the Snapshot was taken.  Again, unless the Snapshot was a cold Snapshot, the VM will likely perform a file system check upon boot.  This is normal and not unexpected.

I suppose you could even do this second procedure from a CIFS client, assuming that CIFS and NFS were both configured on the storage system and an appropriate CIFS share existed.  (Please note that I’ve never tried this, so I can’t tell you what the results might be.)  In that case, use the “~snapshot” directory instead of “.snapshot”.

And that’s it—there you have two ways of recovering entire VMs using Network Appliance Snapshots.  As always, feel free to hit me up in the comments with any questions, thoughts, corrections, or rants (just keep the rants on-topic, please!).  Thanks for reading!

The Sanrad V-Switch is a useful device that offers a number of features, including (but not limited to) storage virtualization, iSCSI-to-FC connectivity, snapshots, and data migration services.  It’s actually a pretty handy little device.  I had the opportunity to spend the day today with a V-Switch 2000, an entry-level model, connected in front of a small Fibre Channel array.  I thought it might be handy to include some configuration commands here in the event that someone else needed them (to be honest, the Sanrad documentation is horrible).

Overall, the process for configuring a V-Switch looks something like this:

1. Physically connect the V-Switch to the Ethernet network and the Fibre Channel fabric(s).
2. Set the IP configuration for the Ethernet interfaces on the V-Switch.
3. Create an iSCSI portal and an iSCSI target on the V-Switch.
4. Present FC storage to the V-Switch.
5. Create volumes on the V-Switch.
6. Present the volumes as iSCSI LUNs.

And that’s it!  Once you get used to the interface (I used the command line interface because…well, I prefer the command line to the GUI).  Let’s walk through those steps real quick.

Physically Connect the V-Switch

The V-Switch gets connected via Gigabit Ethernet and via Fibre Channel.  Physically connecting the V-Switch is as simple as plugging in the Ethernet and Fibre Channel ports to the appropriate switches.  It’s probably also going to be necessary to modify your Fibre Channel zones as well, so that the V-Switch can see any Fibre Channel targets on the fabric(s).

Configure IP

The V-Switch comes with two Gigabit Ethernet interfaces.  To assign IP address(es) to the interface(s), use the ip config set command, like this:

ip config set -ip 10.2.3.45 -if eth1 -im 255.255.255.0

Verify the IP address has been added using ip config show command, as well as by pinging the new IP address from another system on the same subnet.

By the way, the system comes with a pre-assigned IP address (I believe it is 10.11.12.123) which can’t be removed.  At least, I couldn’t figure out how to remove or change it.

Create iSCSI Portal and iSCSI Target

Next, we create an iSCSI portal (IP address and listening TCP port) on the assigned IP address:

iscsi portal create -ip 10.2.3.45 -p 3260

TCP port 3260 is the default, so you could omit that if desired.  The iscsi portal show command will verify the creation of the iSCSI portal.

Once the iSCSI portal is created, create an iSCSI target with the iscsi target create command, like this (I used a backslash to denote line continuation):

iscsi target create -ta vmtarget \
-tn iqn.2000-04.com.sanrad:vswitch01

We can verify the creation of the iSCSI target using iscsi target show.  At this point, iSCSI initiators will see the iSCSI target, but won’t see any LUNs, because we haven’t connected and configured the back-end storage yet.  That’s next.

Present FC Storage to the V-Switch

The exact procedures and commands here will vary based on the back-end storage array.  Create the volumes, disk groups, LUNs, RAID groups, etc., using the standard tools and commands provided by the back-end storage array.  Present those storage containers to the V-Switch using the V-Switch’s WWNN (which can be displayed using the fc interface show command).

Once that’s done, the V-Switch should discover it automatically.  To show the storage that the V-Switch has discovered, use this command:

storage show

This will show disks and controllers with very unimaginative (and undescriptive) names like “Stor_10”.  To keep things logicaly and organized, rename the storage to something that more closely identifies what it is.  In this example, we’ll identify the storage with a logical drive identifier and an array identifier:

storage set -s Stor_15 -na LD09-Ar01 -info Log drv 9, Array 1
storage set -s Stor_6 -na LD00-Ar00 -info Log drv 0, Array 0

With the storage now recognized by the V-Switch and labeled as something recognizable (and traceable back to the storage array itself), we can create a couple of simple volumes:

volume create simple -vol simple00 -d LD00-Ar00
volume create simple -vol simple05 -d LD05-Ar01

Simple volumes are the building blocks for more complex structures like mirrored volumes or striped volumes.  With a couple of simple volumes created, we can create a mirrored volume from these simple volumes:

volume create mirror -vol mirror01 -ch simple00 -ch simple05

Verify the creation of the volumes (simple or mirrored) with the volume show command.

Let’s create a few more simple volumes:

volume create simple -vol simple01 -d LD01-Ar00
volume create simple -vol simple02 -d LD02-Ar00
volume create simple -vol simple06 -d LD06-Ar01
volume create simple -vol simple02 -d LD07-Ar01

Then, using these four simple volumes, let’s create a striped volume:

volume create stripe -vol stripe01 -nbc 4 \
-sus 64 -ch simple01 -ch simple02 -ch simple06 \
-ch simple07

Again, we can use the volume show command to list the volumes that have been created.  However, our job is not yet done.  We have to present these volumes as LUNs to the iSCSI initiators.

Present iSCSI LUNs

The volumes have been created from back-end storage, but now we need to expose the volumes to iSCSI initiators on the iSCSI target created earlier.  We do that with this command:

volume expose -vol mirror01 -ta vmtarget

This exposes the volume named “mirror01” on the iSCSI target named “vmtarget” (created earlier).  That target is accessible on the iSCSI portal also created earlier.

You can verify or show that the volume is exposed as a LUN on the iSCSI target using the lu show command.

Assuming that it is properly exposed, then you should be able to now see the exposed LUN via your iSCSI initiator (software or hardware).

The V-Switch also has some other advanced functionality, like subdisks (carving up LUNs from back-end storage into smaller pieces), snapshots, and data replication/migration.  I hope to get the opportunity to discuss those features more in the future.