The weblog of an IT pro specializing in virtualization, storage, and servers
You are currently browsing articles tagged iSCSI.
I’ve had these FCoE-related articles sitting around in my Yojimbo database for a while, but I’m only now getting around to doing something with them. There’s some great information in these posts, but be sure to check the comments to the posts as well—there’s some equally good information to be found there as well.
FCoE Multi-hop: Why wait?
Re-examining FCoE and iSCSI Pros and Cons
FCoE vs. iSCSI: The Cagefight!
Gartner on FCoE. Whoa There, Sparky
8Gb Fibre Channel or 10Gb Ethernet w/ FCoE?
Tags: FCoE, FibreChannel, iSCSI, Storage
I was browsing through an EMC technical document titled “EMC CLARiiON Integration with VMware ESX Server” (download it here) a little while ago and I came across a phrase in the document that caught my attention:
“VMware ESX/ESXi support both Fibre Channel and iSCSI storage. However, VMware and EMC do not support connecting VMware ESX/ESXi servers to CLARiiON Fibre Channel and iSCSI devices on the same array simultaneously.”
What? No Fibre Channel and iSCSI from the same array to a VMware ESX/ESXi host simultaneously? That piqued my curiosity, so I contacted a few people within EMC to question the veracity of that statement. It turns out that the answer is more complicated than it might seem at first glance.
For those of you who aren’t interested in the deep technical details, here’s the short explanation behind this behavior:
I’m confident that some other array vendors out there will be very quick to jump on this post and harp on this limitation until the cows come home. I would just ask this question: is it really as big of a limitation as it seems? I’ll come back to that question in a moment.
With the short explanation in mind, here are the more in-depth details. If you like the longer, more technical explanation, then read on!
From EMC’s side, the root of the restriction about using both Fibre Channel and iSCSI devices on the same array simultaneously stems from the interaction of host registration and storage groups.
Host registration is a requirement in the CLARiiON world. In order to present storage to a host from a CLARiiON array, you must first register the host’s initiators with the array in Navisphere. Once the host has been registered, then you can proceed with presenting storage to that host. In theory the CLARiiON could operate without registering hosts and initiators, but EMC chose to require registration. EMC made this choice in order to help simplify host management.
Requiring host registration is a bit different than some of other storage arrays on the market. It’s not better or worse—just different. (Remember, pros and cons come from every technology decision.)
If you’re like me, you’re probably wondering at this point how requiring host registration simplifies anything. Instead of having to manage multiple paths, multiple initiators, and individual hosts every time you want to present storage to a host, you only need to register the host—and all of its initiators—and then you can refer to that same object (the host) over and over again as needed. Yes, host registration does mean a bit more work up front, but the idea is that it will save some work down the road. I guess you can think of host registration kind of like defining aliases in your Fibre Channel zoning configuration: it’s a bit more work up front, but it simplifies things later down the road. If you didn’t create device aliases in your Fibre Channel switch, you’d end up having to re-enter Fibre Channel WWPNs multiple times. You create the aliases so that it’s easier later. The same applies to host registration. Again, it’s a matter of choices.
One might also say that registration is security measure, albeit a weak measure. Rather than allow just any Fibre Channel-attached or iSCSI-attached host to see storage, the array requires that it know about the host (via host registration) in order to present storage to the host. This provides an additional layer of security to ensure that only authorized hosts are presented storage from the array.
Now you have a fairly decent idea of why host registration is necessary. So how does host registration occur? Host registration can occur either manually or automatically. Starting with version 4.0, both VMware ESX and VMware ESXi will automatically register with a CLARiiON array running any recent version of FLARE (ESX 3i version 3.5 also supports this form of push registration). FLARE release 28 and earlier will show these hosts as “Manually registered, unmanaged”; starting with FLARE 29, these hosts are listed as “Manually registered, managed”. In either case, the registration occurs automatically. If the host is Fibre Channel-attached, then the Fibre Channel initiators will be included in the automatic registration. The same goes for iSCSI initiators. Normally, this is a good thing because it saves the administrator the extra steps of registering the host with the storage array. (Also, because VMware ESX/ESXi hosts register automatically, there is no need to install the Navisphere Agent.)
In this case, though, the automatic registration causes a problem. Why? This goes back to the second item I said I needed to discuss: storage groups. Specifically, storage groups have two characteristics that come into play here:
Do you see how all the pieces come together? The only way to control which LUNs should be presented via which protocol is to use multiple storage groups—but a host can only be in a single storage group at a time. With only a single host object for any given VMware ESX/ESXi host, that host can only see either Fibre Channel LUNs (by being in a storage group containing Fibre Channel LUNs) or iSCSI LUNs (by being in a storage group containing iSCSI LUNs), but not both. Hence, the statement in the CLARiiON document I referenced in the very beginning of this blog post that outlines using either Fibre Channel or iSCSI but not both. This behavior is required to enforce the single-protocol LUN access required by VMware.
As with all things, there is a workaround. Because it is a workaround, that’s why the RPQ is necessary to get full support.
To work around this problem, you’ll need to ignore the automatic host registration (or disable the automatic host registration) and instead create two manually registered “pseudo-hosts”: one with the Fibre Channel initiators and one with the iSCSI initiators. These “pseudo-hosts” will need fake IP addresses (if they both use the same IP address, Navisphere will treat them as the same host, thus defeating the purpose of the workaround). Put the Fibre Channel initiators into the Fibre Channel storage group(s), and put the iSCSI initiators into the iSCSI storage group(s). Each “pseudo-host” will be able to see LUNs presented to that storage group and therefore would see both Fibre Channel and iSCSI LUNs at the same time. And, as required by VMware, any given LUN would be accessed only via Fibre Channel or iSCSI but not both. Remember that you need to file an RPQ in order to get support on this configuration.
For VMware ESX/ESXi 4.0 hosts (and ESX 3i version 3.5 hosts), you can disable automatic registration using the Disk.EnableNaviReg advanced configuration option. Setting this value to 0 disables the automatic registration with Navisphere. (Here are screenshots for VMware ESX 3i and VMware ESX/ESXi 4.) If you disable the automatic registration, then you only need to manually register the Fibre Channel and iSCSI initiators as separate “pseudo-hosts” and you’re ready to go.
Let me reiterate again that if you are presenting iSCSI LUNs via the Celerra and not the CLARiiON, none of this applies. Presenting Fibre Channel LUNs via the CLARiiON and iSCSI LUNs via the Celerra to the same VMware ESX/ESXi host is fine. This workaround that I’ve described only applies when you want to present some LUNs via Fibre Channel and some LUNs via iSCSI from a CLARiiON to a single VMware ESX/ESXi host.
Earlier you’ll recall that I asked this question: is this really a limitation? There are a couple of viewpoints:
I can see both sides of the coin. Personally, I tend to side more with the second viewpoint and would prefer to see the CLARiiON have the ability to easily present Fibre Channel and iSCSI to the same host, especially when multiple disk pools are involved. I think that CLARiiON engineering is now evaluating this possibility; as more information emerges, I’ll be sure to keep you posted.
Courteous and professional comments, clarifications, or corrections are always welcome!
Tags: EMC, FibreChannel, iSCSI, Storage, Virtualization, VMware
I had a reader contact me with a couple of questions, one of which I felt warranted a blog post. Paraphrased, the question was this: How do I make IP-based storage work with VMware vSphere on Cisco UCS?
At first glance, you might look at this question and scoff. Remember though, that Cisco UCS does—at this time—have a few limitations that make this a bit more complicated than at first glance. Specifically:
With this in mind, then, how does one connect IP-based storage to a Cisco UCS? In these scenarios, you must have another set of Ethernet switches between the 6100XP fabric interconnects and the target storage array. Further, since the 6100XP fabric interconnects require 10GbE uplinks and do not—at this time—offer any 1GbE uplink functionality, you need to have the right switches between the 6100XP fabric interconnects and the target storage array.
Naturally, the Nexus 5000 fits the bill quite nicely. You can use a pair of Nexus 5000 switches between the UCS 6100XP interconnects and the storage array. Dual-connect the 6100XP interconnects to the Nexus 5000 switches for redundancy and active-active data connections, and dual-connect the target storage array to the Nexus 5000 switches for redundancy and (depending upon the array) active-active data connections. It would look something like this:
From the VMware side of the house, since you’re using 10GbE end-to-end, it’s very unlikely that you’ll need to worry about bandwidth; that eliminates any concerns over multiple VMkernel ports on multiple subnets or using multiple NFS targets so as to be able to use link aggregation. (I’m not entirely sure you could use link aggregation with the 6100XP interconnects anyway. Anyone?) However, since you are talking Cisco UCS you’ll have only two 10GbE connections (unless you’re using the full width blade, which is unlikely). This means you’ll need to pay careful attention to the VMware vSwitch (or dvSwitch, or Nexus 1000V) configuration. In general, the recommendation in this sort of configuration is to place Service Console, VMotion, and IP-based storage traffic on one 10GbE uplink, place virtual machine traffic on the second 10GbE uplink, and use whatever mechanisms are available to preferentially specify which uplink should be used in the course of normal operation. This provides redundancy in the uplinks but some level of separation of traffic.
One quick side note: although I’m talking IP-based storage here, block-based storage fans need to remember that Cisco UCS does not—at this time—support northbound FCoE. That means that although you have FCoE support southbound, and FCoE support in the Nexus 5000, and possibly FCoE support in your storage arrays, you still can’t do end-to-end FCoE with Cisco UCS.
For those readers who are very familiar with Cisco UCS and Nexus, this will seem like a pretty simplistic post. However, we need to keep in mind that there are lots of readers out there who have not had the same level of exposure. Hopefully, this will help provide some guidance and food for thought.
(Of course, one could just buy a Vblock and not have to worry about putting all the pieces together…hey, can’t blame me for trying, right?)
Clarifications, questions, or suggestions are welcome in the comments below. Thanks!
Tags: Cisco, iSCSI, NFS, UCS, Vblock, Virtualization, VMware, vSphere
Welcome to Virtualization Short Take #30, my irregularly posted collection of links and thoughts on virtualization. I hope you find something useful here!
That wraps it up for this time around. Thanks for reading, and feel free to submit any other useful or interesting links in the comments below.
Tags: ESX, ESXi, iSCSI, Virtualization, VMware, VMwareFT, vSphere
I wanted to go ahead and get another issue of Virtualization Short Takes out the door before VMworld, as I suspect that I’ll be covered up both during and for some time after VMworld. So, here’s my latest collection of links and articles about virtualization, storage, and anything else I find interesting.
That wraps up this edition of Virtualization Short Takes. Next week will be a busy week; look for lots of coverage from the conference in San Francisco as well as summaries of my vendor meetings (and there are lots of them!). Until then, take care!
Tags: EMC, iSCSI, Microsoft, NetApp, Storage, Virtualization, VMware, vSphere
In April 2008, I wrote an article on how to use jumbo frames with VMware ESX and IP-based storage (NFS or iSCSI). It’s been a pretty popular post, ranking right up there with the ever-popular article on VMware ESX, NIC teaming, and VLAN trunks.
Since I started working with VMware vSphere (now officially available as of 5/21/2009), I have been evaluating how to replicate the same sort of setup using ESX/ESXi 4.0. For the most part, the configuration of VMkernel ports to use jumbo frames on ESX/ESXi 4.0 is much the same as with previous versions of ESX and ESXi, with one significant exception: the vNetwork Distributed Switch (vDS, what I’ll call a dvSwitch). After a fair amount of testing, I’m pleased to present some instructions on how to configure VMkernel ports for jumbo frames on a dvSwitch.
The lab configuration for this testing was pretty straightforward:
Keep in mind that these procedures or commands may be different in your environment, so plan accordingly.
Refer back to my first article on jumbo frames to review the Cisco IOS commands for configuring the physical switch to support jumbo frames. Once the physical switch is ready to support jumbo frames, you can proceed with configuring the virtual environment.
The virtual network configuration consists of several steps. First, you must configure the dvSwitch to support jumbo frames by increasing the MTU. Second, you must create a distributed virtual port group (dvPort group) on the dvSwitch. Finally, you must create the VMkernel ports with the correct MTU. Each of these steps is explained in more detail below.
Setting the MTU on the dvSwitch is pretty straightforward:
That’s it! Now, if only the rest of the process was this easy…
By the way, this same area is also where you can enable Cisco Discovery Protocol support for the dvSwitch, as I pointed out in this recent article.
Like setting the MTU on the dvSwitch, this process is pretty straightforward and easily accomplished using the vSphere Client:
See? I told you it was pretty straightforward. Now on to the final step which, unfortunately, won’t be quite so straightforward or easy.
Now things get a bit more interesting. As of the GA code, the vSphere Client UI still does not expose an MTU setting for VMkernel ports, so we are still relegated to using the esxcfg-vswitch command (or the vicfg-vswitch command in the vSphere Management Assistant—or vMA—if you are using ESXi). The wrinkle comes in the fact that we want to create a VMkernel port attached to a dvPort ID, which is a bit more complicated than simply creating a VMkernel port attached to a local vSwitch.
Disclaimer: There may be an easier way than the process I describe here. If there is, please feel free to post it in the comments or shoot me an e-mail.
First, you’ll need to prepare yourself. Open the vSphere Client and navigate to the Hosts and Clusters inventory view. At the same time, open an SSH session to one of the hosts you’ll be configuring, and use “su -” to assume root privileges. (You’re not logging in remotely as root, are you?) If you are using ESXi, then obviously you’d want to open a session to your vMA and be prepared to run the commands there. I’ll assume you’re working with ESX.
This is a two-step process. You’ll need to repeat this process for each VMkernel port that you want to create with jumbo frame support.
Here are the steps to create a jumbo frames-enabled VMkernel port:
From this point on you’ll go the rest of the way from the command line. However, leave the Manage Virtual Adapters dialog box open and the vSphere Client running.
To finish the process from the command line:
esxcfg-vswitch -lesxcfg-vmknic -d -s <dvSwitch Name> -v <dvPort ID>esxcfg-vmknic -a -i <IP addr> -n <Mask> -m 9000 -s <dvSwitch Name> -v <dvPort ID>At this point, you can go back into the vSphere Client and enable the VMkernel port for VMotion or FT logging. I’ve tested jumbo frames using VMotion and everything is fine; I haven’t tested FT logging with jumbo frames as I don’t have FT-compatible CPUs. (Anyone care to donate some?)
As I mentioned in yesterday’s Twitter post, I haven’t conducted any objective performance tests yet, so don’t ask. I can say that NFS feels faster with jumbo frames than without, but that’s purely subjective.
Let me know if you have any questions or if anyone finds a faster or easier way to accomplish this task.
UPDATE: I’ve updated the comments to delete and recreate the VMkernel port per the comments below.
Tags: CLI, ESX, ESXi, iSCSI, Networking, NFS, Storage, Virtualization, VMware, vSphere
This session describes NetApp’s MultiStore functionality. MultiStore is the name given to NetApp’s functionality for secure logical partitioning of network and storage resources. The presenters for the session are Roger Weeks, TME with NetApp, and Scott Gelb with Insight Investments.
When using MultiStore, the basic building block is the vFiler. A vFiler is a logical construct within Data ONTAP that contains a lightweight instance of the Data ONTAP multi-protocol server. vFilers provide the ability to securely partition both storage resources and network resources. Storage resources are partitioned at either the FlexVol or Qtree level; it’s recommended to use FlexVols instead of Qtrees. (The presenters did not provide any further information beyond that recommendation. Do any readers have more information?) On the network side, the resources that can be logically partitioned are IP addresses, VLANs, VIFs, and IPspaces (logical routing tables).
Some reasons to use vFilers would include storage consolidation, seamless data migration, simple disaster recovery, or better workload management. MultiStore integrates with SnapMirror to provide some of the functionality needed for some of these use cases.
MultiStore uses vFiler0 to denote the physical hardware, and vFiler0 “owns” all the physical storage resources. You can create up to 64 vFiler instances, and active/active clustered configurations can support up to 130 vFiler instances (128 vFilers plus 2 vFiler0 instances) during a takeover scenario.
Each vFiler stores its configuration in a separate FlexVol (it’s own root vol, if you will). All the major protocols are supported within a vFiler context: NFS, CIFS, iSCSI, HTTP, and NDMP. Fibre Channel is not supported; you can only use Fibre Channel with vFiler0. This is due to the lack of NPIV support within Data ONTAP 7. (It’s theoretically possible, then, that if/when NetApp adds NPIV support to Data ONTAP that Fibre Channel would be supported within vFiler instances.)
Although it is possible to move resources between vFiler0 and a separate vFiler instance, doing so may impact client connections.
Managing vFilers appears to be the current weak spot; you can manage vFiler instances using the Data ONTAP CLI, but vFiler instances don’t have an interactive shell. Therefore, you have to direct commands to vFiler instances via SSH or RSH or using the vFiler context in vFiler0. You access the vFiler context by prepending the “vfiler” keyword to the commands at the CLI in vFiler0. Operations Manager 3.7 and Provisioning Manager can manage vFiler instances; FilerView can start, stop, or delete individual vFiler instances but cannot direct commands to an individual vFiler. If you need to manage CIFS on a vFiler instance, you can use the Computer Management MMC console to connect remotely to that vFiler instance to manage shares and share permissions, just as you can with vFiler0 (assuming CIFS is running within the vFiler, of course).
IPspaces are a logical routing construct that allow each vFiler to have its own routing table. For example, you may have a DMZ vFiler and an internal vFiler, each with their own, separate routing table. Up to 101 IPspaces are supported per controller. You can’t delete the default IPspace, as it’s the routing table for vFiler0. It is recommended to use VLANs and/or VIFs with IPspaces as a best practice.
One of the real advantages of using MultiStore and vFilers is the data migration and disaster recovery functionality that it enables when used in conjunction with SnapMirror. There are two sides to this:
It makes sense, but you can’t use “vfiler dr” or “vfiler migrate” on vFiler0 (the physical storage system). My own personal thought regarding “vfiler dr”: what would this look like in a VMware environment using NFS? There could be some interesting possibilities there.
With regard to security, a Matasano security audit was performed and the results showed that there were no vulnerabilities that would allow “data leakage” between vFiler instances. This means that it’s OK to run a DMZ vFiler and an internal vFiler on the same physical system; the separation is strong enough.
Other points of interest:
Some of the potential use cases for MultiStore include file services consolidation (allows you to preserve file server identification onto separate vFiler instances), data migration, and disaster recovery. You might also use MultiStore if you needed support for multiple Active Directory domains with CIFS.
UPDATE: Apparently, my recollection of the presenters’ information was incorrect, and FTP is not a protocol supported with vFilers. I’ve updated the article accordingly.
Tags: ActiveDirectory, CLI, Deduplication, Insight2008, iSCSI, NetApp, NFS, ONTAP, SAN, Storage, Virtualization, VMware, WAFL
So here’s another “thinking out loud” post. This time, I’m thinking about Fibre Channel over Ethernet (FCoE) and unified fabric.
I was going back through a list of blog posts and articles that I wanted to read and think on, and I came across a link to Dave Graham’s article titled Moving a Fabric forward: FCoE Adoption and other Questions. His blog entry was partially in response to my FCoE discussion post. His post got me thinking again.
It seems like anytime someone talks about FCoE, they end up also talking about unified fabric. After having read a number of different articles and posts regarding FCoE, I can see where FCoE would be attractive to shops with significant FCP installations. In my mind, though, this doesn’t necessarily mean unified fabric. Given the political differences in organizations—think the “storage team” and the “networking team”—how likely is it that an organization may adopt FCoE, but not unified fabric? Or how likely is it that an organization may adopt FCoE, intending it to be a transitional technology leading to unified fabric, but never actually make it all the way?
So here’s my question: is unified fabric an inevitability?
(Oh, and here’s a related question: Most people cite VoIP as proof that the unified fabric is inevitable. More so than anything else, I believe VoIP’s success was more a reflection of the rising importance of TCP/IP networking. If so, does that give iSCSI an edge over FCoE? Is iSCSI the “VoIP of the storage world”?)
Tags: FCoE, FibreChannel, iSCSI, Storage
Via this Twitter post, I learned that VMware has posted a new KB article about a SAN I/O failure that can occur with VMware ESX and ESXi 3.5 Update 3. If you are using VMware ESX or ESXi 3.5 Update 3, please be sure to read the KB article and stay tuned here for additional updates. I will update this post when I get more information.
Tags: ESX, ESXi, FibreChannel, iSCSI, SAN, Storage, Virtualization, VMware
There’s lots of good information flowing around the Internet, and it’s becoming increasingly difficult to sort through all the useless stuff to find the valuable gems. Hopefully, some of the links that I have collected here will prove to be more useful than useless!
It’s clearly been too long since I published one of these, as I still have other links collecting dust in my “link bin”:
Third Brigade offers free security for up to 100 virtual machines
Version 4 of the PowerVDI tool
Go Daddy Wildcard Certificate with VI3
New VMware VI network port diagram request for comments
Auditing ESX root logins with email…
Like I said, there’s just so much information! And now that I’m trying to delve deeper into the storage realm, that’s only doubled up on the information I’m trying to manage. Hopefully I’ve picked out a few gems for you this week. Thanks for reading!
Tags: ActiveDirectory, EMC, ESX, HyperV, iSCSI, NFS, SSL, VMotion, VMware, VMwareSRM
Recent Bookmarks
Scott’s Weblog by Scott Lowe is licensed under a Creative Commons Attribution-Share Alike 3.0 United States License.
Recent Comments