VMware ESX, EMC CLARiiON Arrays, and Multiple Protocols

I was browsing through an EMC technical document titled “EMC CLARiiON Integration with VMware ESX Server” (download it here) a little while ago and I came across a phrase in the document that caught my attention:

“VMware ESX/ESXi support both Fibre Channel and iSCSI storage. However, VMware and EMC do not support connecting VMware ESX/ESXi servers to CLARiiON Fibre Channel and iSCSI devices on the same array simultaneously.”

What? No Fibre Channel and iSCSI from the same array to a VMware ESX/ESXi host simultaneously? That piqued my curiosity, so I contacted a few people within EMC to question the veracity of that statement. It turns out that the answer is more complicated than it might seem at first glance.

For those of you who aren’t interested in the deep technical details, here’s the short explanation behind this behavior:

  • VMware fully supports the use of both Fibre Channel and iSCSI from the same array to the same VMware ESX/ESXi host simultaneously.
  • VMware does not support presenting the same LUN via both protocols concurrently to the same host. (I qualified this directly with VMware.)
  • For a Celerra, you can use both Fibre Channel (via the CLARiiON side of the array) and iSCSI (via the Celerra side of the array) simultaneously. This is a fully supported configuration.
  • A CLARiiON array can easily present the same LUN via both Fibre Channel and iSCSI, but then VMware wouldn’t support it (see earlier bullet).
  • With a CLARiiON array, it is possible to present some LUNs via Fibre Channel and some LUNs via iSCSI to the same VMware ESX/ESXi host (i.e., LUN A via Fibre Channel and LUN B via iSCSI), but EMC will only support it if you file an RPQ. Without an RPQ, it’s an unsupported configuration. An RPQ, by the way, is a request to qualify a certain configuration for support.

I’m confident that some other array vendors out there will be very quick to jump on this post and harp on this limitation until the cows come home. I would just ask this question: is it really as big of a limitation as it seems? I’ll come back to that question in a moment.

With the short explanation in mind, here are the more in-depth details. If you like the longer, more technical explanation, then read on!

From EMC’s side, the root of the restriction about using both Fibre Channel and iSCSI devices on the same array simultaneously stems from the interaction of host registration and storage groups.

Host registration is a requirement in the CLARiiON world. In order to present storage to a host from a CLARiiON array, you must first register the host’s initiators with the array in Navisphere. Once the host has been registered, then you can proceed with presenting storage to that host. In theory the CLARiiON could operate without registering hosts and initiators, but EMC chose to require registration. EMC made this choice in order to help simplify host management.

Requiring host registration is a bit different than some of other storage arrays on the market. It’s not better or worse—just different. (Remember, pros and cons come from every technology decision.)

If you’re like me, you’re probably wondering at this point how requiring host registration simplifies anything. Instead of having to manage multiple paths, multiple initiators, and individual hosts every time you want to present storage to a host, you only need to register the host—and all of its initiators—and then you can refer to that same object (the host) over and over again as needed. Yes, host registration does mean a bit more work up front, but the idea is that it will save some work down the road. I guess you can think of host registration kind of like defining aliases in your Fibre Channel zoning configuration: it’s a bit more work up front, but it simplifies things later down the road. If you didn’t create device aliases in your Fibre Channel switch, you’d end up having to re-enter Fibre Channel WWPNs multiple times. You create the aliases so that it’s easier later. The same applies to host registration. Again, it’s a matter of choices.

One might also say that registration is security measure, albeit a weak measure. Rather than allow just any Fibre Channel-attached or iSCSI-attached host to see storage, the array requires that it know about the host (via host registration) in order to present storage to the host. This provides an additional layer of security to ensure that only authorized hosts are presented storage from the array.

Now you have a fairly decent idea of why host registration is necessary. So how does host registration occur? Host registration can occur either manually or automatically. Starting with version 4.0, both VMware ESX and VMware ESXi will automatically register with a CLARiiON array running any recent version of FLARE (ESX 3i version 3.5 also supports this form of push registration). FLARE release 28 and earlier will show these hosts as “Manually registered, unmanaged”; starting with FLARE 29, these hosts are listed as “Manually registered, managed”. In either case, the registration occurs automatically. If the host is Fibre Channel-attached, then the Fibre Channel initiators will be included in the automatic registration. The same goes for iSCSI initiators. Normally, this is a good thing because it saves the administrator the extra steps of registering the host with the storage array. (Also, because VMware ESX/ESXi hosts register automatically, there is no need to install the Navisphere Agent.)

In this case, though, the automatic registration causes a problem. Why? This goes back to the second item I said I needed to discuss: storage groups. Specifically, storage groups have two characteristics that come into play here:

  1. First, any given host—not just VMware ESX/ESXi hosts, but all types of hosts—can only be connected to a single storage group at any given time.
  2. Second, while the CLARiiON can present Fibre Channel LUNs and iSCSI LUNs simultaneously (including presenting the same LUN via both protocols simultaneously), there is no way within a single storage group to specify which LUNs should be accessed via Fibre Channel and which LUNs should be accessed via iSCSI. This is necessary because VMware won’t support accessing the same LUN via both protocols at the same time (see earlier VMware support statement).

Do you see how all the pieces come together? The only way to control which LUNs should be presented via which protocol is to use multiple storage groups—but a host can only be in a single storage group at a time. With only a single host object for any given VMware ESX/ESXi host, that host can only see either Fibre Channel LUNs (by being in a storage group containing Fibre Channel LUNs) or iSCSI LUNs (by being in a storage group containing iSCSI LUNs), but not both. Hence, the statement in the CLARiiON document I referenced in the very beginning of this blog post that outlines using either Fibre Channel or iSCSI but not both. This behavior is required to enforce the single-protocol LUN access required by VMware.

As with all things, there is a workaround. Because it is a workaround, that’s why the RPQ is necessary to get full support.

To work around this problem, you’ll need to ignore the automatic host registration (or disable the automatic host registration) and instead create two manually registered “pseudo-hosts”: one with the Fibre Channel initiators and one with the iSCSI initiators. These “pseudo-hosts” will need fake IP addresses (if they both use the same IP address, Navisphere will treat them as the same host, thus defeating the purpose of the workaround). Put the Fibre Channel initiators into the Fibre Channel storage group(s), and put the iSCSI initiators into the iSCSI storage group(s). Each “pseudo-host” will be able to see LUNs presented to that storage group and therefore would see both Fibre Channel and iSCSI LUNs at the same time. And, as required by VMware, any given LUN would be accessed only via Fibre Channel or iSCSI but not both. Remember that you need to file an RPQ in order to get support on this configuration.

For VMware ESX/ESXi 4.0 hosts (and ESX 3i version 3.5 hosts), you can disable automatic registration using the Disk.EnableNaviReg advanced configuration option. Setting this value to 0 disables the automatic registration with Navisphere. (Here are screenshots for VMware ESX 3i and VMware ESX/ESXi 4.) If you disable the automatic registration, then you only need to manually register the Fibre Channel and iSCSI initiators as separate “pseudo-hosts” and you’re ready to go.

Let me reiterate again that if you are presenting iSCSI LUNs via the Celerra and not the CLARiiON, none of this applies. Presenting Fibre Channel LUNs via the CLARiiON and iSCSI LUNs via the Celerra to the same VMware ESX/ESXi host is fine. This workaround that I’ve described only applies when you want to present some LUNs via Fibre Channel and some LUNs via iSCSI from a CLARiiON to a single VMware ESX/ESXi host.

Earlier you’ll recall that I asked this question: is this really a limitation? There are a couple of viewpoints:

  • One viewpoint states there is no need for both Fibre Channel and iSCSI connectivity to the same array. Since you already have Fibre Channel connectivity to the array, what’s the point in using iSCSI? Conversely, if you already have iSCSI connectivity to an array, why invest in establishing Fibre Channel connectivity? Since you can’t use it for failover (that would violate the VMware support position), running another block protocol against the same array and same sets of disks doesn’t add a great deal of value.
  • A second viewpoint argues that the ability to provide a differentiation of service based on the different performance characteristics of Fibre Channel and iSCSI (and NFS, but we’re focusing on block protocols for this discussion) is valuable, and thus the need to be able to easily present LUNs via either protocol from the same array to the same host is a worthwhile function. There are a number of potential use cases here—test/development environments, Tier 2 applications, varying SLAs, etc. This is especially true if you are using different disk pools (fast Fibre Channel drives or EFDs vs. slower SATA drives) on the same array.

I can see both sides of the coin. Personally, I tend to side more with the second viewpoint and would prefer to see the CLARiiON have the ability to easily present Fibre Channel and iSCSI to the same host, especially when multiple disk pools are involved. I think that CLARiiON engineering is now evaluating this possibility; as more information emerges, I’ll be sure to keep you posted.

Courteous and professional comments, clarifications, or corrections are always welcome!

Tags: , , , , ,

15 comments

  1. Jason Boche’s avatar

    Fabric Aliasing is a good analogy for HBA initiator registration. I see the long term benefits of registration once that up front work is completed and registration for ESX(i) hosts is farily painless. My wish is that registration was as easy (and agentless) for other host platforms. It would be nice if the agent was baked into the HBA firmware somehow.

    Please let me know if I’m off base or misunderstanding anything here. I haven’t done a lot of agent registrations, only a few ESX(i), one Windows, and an NPIV VM.

    Jas

  2. Itzik reich’s avatar

    Another use case for using different protocols is for vdi bars solutions, you may want to put the user data on NFS stores then dedupe these stores, same applies for the user apps, you mayeant to put them on iSCSI
    lastly, with the coming plugin…

    Itzik Reich

  3. Craig’s avatar

    I just completed 1 of the installation form EMC CX4-120 with Cisco UCS. The EMC guys updated with the latest flare code, I dunno what edition is that, forgot to ask, but the host registration process is automated. Whenever my server get connected to the clariion, from navisphere I will able to see the physical host been automated registered with the FQDN, IP address and WWN address. Compare to the previous flare code, we required to manually register each of the physical host from navisphere. Again, it does not take too much effort for us.

  4. Vijay Swami’s avatar

    Scott, what *specific* use cases were you think where this would be useful? I’m trying to think of any where it would be handy to serve iSCSI and FC from the a single ESX host but can’t think of any, nor can recall any with current/previous customer’s where it would have came in handy. Can you elaborate on some specific scenarios you had in mind? Typically customer’s like to segment off test/dev environments into their own HA/DRS clusters so they are completely fenced from production. Even with the advent of resource pools, I see advantages to a separate dev/test cluster in many customer environments. So my point is, while I think it’s a nice feature, I just can’t seem to come up with any real world applications.

    Thoughts?

  5. Evaldas’s avatar

    Scott, we actually use the same LUN via multiprotocol-multipath to ESX from OpenSolaris COMSTAR targets, FC and iSCSI (1G/10G), basically for failover, I think you could call EMC’s host registration as “views” in COMSTAR, there you can take initiator identifiers wwn.., iqn.. and put them in a group and then create a view for that group on a particual LUN (Mike LaSpina has a described it in very nice details here: http://blog.laspina.ca/ubiquitous/multi_protocol_storage_provisioning_with ). Here is “Manage paths” screenshot for one of our LUNs – http://zaibasweb.nerim.net/esx-multiprotocol-multipath.png. The only annoyance with this setup is that VC “Storage Views” tab is not working. Do you have an idea if VMware has any plans on supporting multiproto for same LUN, working in that direction ?

  6. Vinod Pant’s avatar

    I am trying to create multipathing between Clariion CX3-10 and ESX4, but have not got the success so far. It detects the path 5 which is correct but shows only one path under “managed path”. What I did so far is, create multiple vmkernel port groups, assigned each port group a dedicated NIC and then bind these vmk port groups on the console.
    I am using software iSCSI intiator at ESX4 and iSCSI connection from CX3-10.

    Is there any doc available in the internet that could explained the process? I have not found any.
    Thanks,
    V

  7. John Philips’s avatar

    I have several ESXi 4.0 hosts connected to a CX4 SAN. Some hosts connect with iSCSI, some with FC, some with both. For the hosts with both, the “Manage Paths…” portion of the vCenter GUI shows that the same datastores are accessible via iSCSI and FC paths. The SAN is still running v28 software.

    Maybe it’s not possible to use multipath/load balancing over iSCSI & FC at the same time, but at least for failover purposes it looks do-able out-of-the-box with no special configuration.

  8. slowe’s avatar

    John Philips,

    It’s absolutely possible, as you’ve so clearly discovered. It’s just not SUPPORTED. VMware does not support accessing the same LUN over both FC and iSCSI.

  9. David Hesse’s avatar

    very good article. Recently I had a discussion with one of my colleagues about that exact topic and we had different opionions on what is possible and what not. Your article has tackled the mystery. Thank you :-)

  10. InsaneGeek’s avatar

    So, FC & iscsi the same host is unsupported… any idea if having some hosts using FC & some using iscsi to the same lun is supported in a HA cluster (a host is either all FC or iscsi not both)?

  11. slowe’s avatar

    InsaneGeek,

    Small clarification: it’s not that FC and iSCSI on the same host isn’t supported; you could do FC to one array and iSCSI to a separate array with no problems. It’s FC and iSCSI on the same host to the same array that isn’t supported by EMC, and FC and iSCSI to the same LUN simultaneously that isn’t supported by VMware.

    As for a cluster, FC on some hosts and iSCSI on some hosts to the same array should be fine. Not necessarily a recommended configuration, but it should work.

  12. Mark’s avatar

    Hi,
    I am having the same problem. I have 1 host with 2 x HBA ports. I want to access Lun0 down path 1 and Lun1 down path 2. Now when my host registers with the clariion it adds both pwwns to the same host name as it recognizes the same fqdn and IP address. So now I have this Host1 >pwwn1 and pwwn2.
    Now I cannot separate access down each path to the luns as I only have 1 host with 2 x pwwns under it. I want 2 hosts with 1 pwwn under each so I can create two storage groups with one pwwn under each.
    How can this be done? by manual registration and fake IP addresses only?

  13. slowe’s avatar

    Mark,

    My question is why do you want to have this setup? What is driving you to say that one LUN should be visible on one HBA and a different LUN visible on a different HBA? Depending on why you want to have this sort of design, there might be a better way of approaching the problem.

  14. Anil’s avatar

    Hi,
    I need some clarification. I want to configure FC and ISCSI on same storage may be different Arrays. I want to use ISCSI for VM Installation and FC for Database in that VM. Is it Possible.

  15. slowe’s avatar

    Anil, without some additional information on your configuration I don’t think it’s really possible to provide some concrete feedback. I can tell you that, generally speaking, you won’t do FC storage directly from a VM, as most virtualization solutions can’t pass an FC HBA directly to the guest. (Further, for those solutions that can, it introduces significant limitations in the design.) If you want to use in-guest iSCSI (iSCSI traffic originating out of the guest VM, not the host), that is something that is supported and will work with most virtualization solutions and most storage arrays. Good luck!

Comments are now closed.