You are currently browsing articles tagged SAN.

This is BRKSAN-3707, Advanced SAN Services, presented by Mike Dunn.

According to TIP’s Storage Research, the top five storage initiatives are deduplication, technology refresh, tiered storage build-out, archiving, and consolidation.

The three main sections of the presentation are SAN consolidation with virtualization, tiered storage and backup design, and Fibre Channel over Ethernet (FCoE).

The presentation starts with SAN consolidation using virtualization. This is really a discussion of virtual SANs (VSANs), which allow you to consolidate SANs onto the same hardware but still providing logical separation of fabrics. In order to move between VSANs, you need to use Inter-VSAN Routing (IVR).

When is IVR needed? When an initiator in one VSAN needs to talk to a target in another VSAN. IVR maintains isolation while allowing for resource sharing.

A common use for IVR is to provide common SAN services, like a shared tape library. IVR would allow media servers in individual VSANs to talk to a shared tape library in a “common” VSAN.

Setting up IVR involves creating an IVR topology. This means you need to manually define the VSANs that will be used for IVR on each switch (all switches that perform IVR will need identical configuration). After defining the IVR topology, you activate it. Then you create your IVR zones and IVR zoneset, just like creating regular zones and zoneset.

IVR works by creating a virtual domain in each VSAN that represents the other VSANs in the topology. Likewise, it creates virtual devices in each VSAN that represent the devices in the other VSANs. This means that logically the initiator thinks the target is in the same VSAN.

Keep in mind that IVR doesn’t perform FC ID translation, so domain IDs have to be unique across all VSANs.

IVR does have a Network Address Translation (NAT) mode (IVR NAT). With IVR NAT, the virtual switch is given a randomly available domain ID; this means that you don’t need unique domain IDs across all VSANs. IVR NAT is the preferred method of IVR moving forward.

Some operating systems or devices need persistent FC IDs, so IVR NAT allows for static definitions of domain IDs and FC IDs.

Another use case for IVR is SAN extension. You can use IVR to isolate the “remote site” VSAN from the production VSAN, limiting edge VSAN events to only that VSAN. The recommended configuration uses a transit VSAN that connects the two data centers. This keeps the VSANs in each data center isolated to only that data center. (Think of it like a /30 network between two routers.)

A question was asked about using FCIP and whether IVR is needed in this sort of situation. In this case, IVR would not be necessary.

Mike next launches into a quick review of SAN designs. In a core-edge design, there are core switches where storage is attached and edge switches where hosts are attached. This sort of design generally tops out at about 1,700 devices.

For larger environments, you can use an edge-core-edge design. Storage devices have their own edge switches, as do servers, and the edge-to-edge traffic passes through the core. This sort of design tops out at about 4,200 devices.

That discussion was a lead-in to a discussion of NPV/NPIV. This is a topic I’ve covered previously, so I didn’t take notes on this section.

Mike did share some good information on the maximum number of logins per port (42 in switch mode, 114 in NPV mode—watch this value if you are using nested NPIV, which is the term for NPIV hosts connecting to an NPV mode switch) and logins per switch (MDS 9124/9124e/9134/9148).

After discussing NPV/NPIV, Mike moves on to discuss a feature called FlexAttach. FlexAttach resolves the issue of needing to modify zoning and zonesets when an HBA or server needs to be replaced. Basically, any host connecting to an F-port configured as a FlexAttach port will assume the virtual WWPN assigned to that F-port. This eliminates the need to reconfigure zones or zonesets if you replace the server or HBA connected to that F-port. If you’re familiar with the behavior of HP VirtualConnect, this appears to be very similar in behavior. FlexAttach is supported on the MDS platform, but is not supported on the Nexus platform.

(Side question: Is FlexAttach leveraged in UCS for vHBA configurations?)

That wraps up the first section; Mike now moves into a discussion of tiered storage and backup design. In this section he will discuss Data Mobility Manager (DMM), SANTap, and Storage Media Encryption (SME).

To perform data migrations, there are different approaches:

  • Server/software-based
  • Array-based
  • Appliance-based

Each of these approaches has advantages and disadvantages. Cisco’s solution is DMM, which is a SAN-based migration solution. DMM does both online and offline data migration, uses FC redirects to allow transparent insertion/removal, and is very fast (4.2TB/hr).

FC Redirect is a target-based mechanism for transparently redirecting traffic to a target. With regard to DMM specifically, when using DMM for data migration, FC redirect is used to redirect traffic to the DMM process itself. DMM then sends the I/Os to both the original (source) and destination locations on the SAN. In this regard, it sounds like DMM is performing a form of write-splitting.

In synchronous mode, when handling I/Os to a “migrated” area of the LUN, writes are mirrored. If I/Os are to a “in process” area, writes are queued temporarily until the region has been migrated. For I/Os to “unmigrated” areas are simply sent directly to the source LUN.

In a dual-fabric configuration, each fabric requires its own DMM. Each DMM can handle multiple VSANs.

DMM can also run in an asynchronous mode. In this mode, DMM uses Modified Region Logs (MRLs) to track changes to the source LUN. Any “dirty region” in the MRL is copied across to the target. There is no write penalty as there is with synchronous mode described earlier.

A question was raised about what happens when the data migration is complete. At that point, you’ll halt the I/O on the server, complete the job in DMM, and then rezone the fabric to point your host(s) to the new storage target.

You can use the DMM asynchronous mode to migrate data between data centers as well. To prevent having to span a VSAN to the remote site (generally not recommended), you can add another VSAN (a replication VSAN) and a third MSM (a module in the FC switch that runs DMM) to handle the inter-site traffic.

The 120 day evaluation licenses within NX-OS will enable DMM with full functionality for 120 days.

The presentation next shifts to Storage Media Encryption (SME). SME encrypts media for SAN-attached tapes, VTLs, and disk arrays. It uses AES-256 encryption and it is FIPS 140-2 certified. The solution can use a Cisco key management solution or RSA Key Manager. SME is a licensed feature and is only supported on certain platforms (requires certain modules, i.e., the MSM-18/4 or the SSN16).

SME uses FC redirects to transparently insert itself into the data stream to perform encryption.

Cisco’s key management solution, Key Management Center, is part of Cisco Fabric Manager. It handles archiving, replicating, recovering, and purging media keys.

Encrypting disks using SME will be available in NX-OS 5.2(1).

The next topic up is SANTap, which as many readers already know is leveraged by EMC RecoverPoint for heterogeneous storage replication. SANTap is a licensed feature but does not use FC redirects. Instead, SANTap uses a host VSAN and a target VSAN. In the host VSAN, SANTap creates a DVT (Data Virtual Target), which uses the WWPN of the real target port. In the target VSAN, SANTap creates a VI (Virtual Initiator), which uses the WWPN of the real host port. SANTAp issues I/Os (or splits I/Os) from the host to the DVT and passes a copy to an additional fabric-based appliance (i.e., a RecoverPoint appliance).

Mike did not have any information on SANTap support for the SCSI commands used by VMware for the VAAI/VAAIv2 offloads in vSphere 4 and vSphere 5. (Bummer!)

The last section of the presentation was on Fibre Channel over Ethernet (FCoE). The information contained in this section was review and stuff that I’ve already covered elsewhere.

Tags: , , , , ,

Chris M Evans (@chrismevans) aka The Storage Architect recently posted an article on his site titled “VAAI – Offloading or Maintaining Control?” The article was a follow-up to a discussion with Nigel Poulton (@nigelpoulton) at HP Discover 2011.

The basic premise of the article as I understand it was a discussion of whether the introduction of VAAI (vStorage APIs for Array Integration), while it does bring benefits to the storage layer, also gives VMware more control. Quoting from the article:

In comparison to NFS, it seems that by implementing VAAI features, VMware are looking to retain even more control over the storage layer than before.

At first glance, it would seem that this conclusion is logical: VMware is using its popularity and large market share to force storage vendors to write to their proprietary “array integration APIs”, thus giving them another layer of control over the storage layer. VMware is the one pushing these integrations, so it seems like a reasonable conclusion, right?

The problem with this conclusion is that it overlooks one simple fact: the vStorage APIs for Array Integration aren’t APIs at all; in fact, they are nothing more than standard SCSI commands. That’s right: VAAI is composed of standard SCSI commands that have been defined and approved by the INCITS T10 committee. (If you’re interested, you can get details on the commands in the SBC-3 standard available from the T10 website.) Any hypervisor or OS running any file system on any array could leverage these commands and their functionality. There’s absolutely nothing specific to VMware or VMFS, VMware’s proprietary clustered file system, that would grant VMware any level of control. Microsoft could add support for these T10 commands and Hyper-V would reap the same benefits as VMware. The open source community could add support for these commands to Xen or KVM, if they so desired, and they would receive the same benefits that vSphere receives.

The flip side is true, too: there’s nothing in any of these SCSI commands that grants a storage vendor a level of control, either. NetApp, EMC, HP, HDS, Dell/Compellent/EqualLogic—anyone of these vendors could (and many did) implement the commands and would support them regardless of the software running on the host.

In my mind, the fact that these commands are completely hypervisor- and OS–agnostic (meaning that any hypervisor or OS could implement them) completely deflates the theory that VAAI gives VMware another form of control at the storage layer.

What do you think? Speak up in the comments.

Tags: , , , ,

I’m so far behind in my technology reading that I have this massive list of blog posts and links that I would normally put into an issue of Technology Short Takes. However, people are already “complaining” that my Short Takes aren’t all that short. To keep from overwhelming people, I’m breaking Technology Short Take #12 into three editions: Virtualization, Storage, and Networking.

Here’s the “Storage Edition” of Technology Short Take #12!

  • When planning storage details for your vSphere implementation, be sure to keep block size in mind. Duncan Epping’s post on the performance impact of the different datamovers in a Storage vMotion operation should bring to light why this is an important storage detail to remember. (And read this post if you need more info on the different datamovers.)
  • Richard Anderson of EMC (aka @storagesavvy) posted a “what if” about using cloud storage as a buffer with thin provisioning and FAST VP. It’s an interesting idea, and one that will probably see greater attention moving forward.
  • Richard also shared some real-world results on the benefits of using FAST Cache and FAST VP on a NS-480 array.
  • Interested in using OpenFiler as an FC target? Have a look here.
  • Nigel Poulton posted an analysis of EMC’s recent entry in the SPC benchmarketing wars in which he compares storage benchmarking to Formula 1 racing. I can see and understand his analogy, and to a certain extent he has a valid point. On the other hand, it doesn’t make sense to submit a more “mainstream” configuration if it’s a performance benchmark; to use Nigel’s analogy, that would be like driving your mini-van in a Formula 1 race. Yes, the mini-van is probably more applicable and useful to a wider audience, but a Formula 1 race is a “performance benchmark,” is it not? Anyway, I don’t know why certain configurations were or were not submitted; that’s for far more important people than me to determine.
  • Vijay (aka @veverything on Twitter) has a good deep dive on EMC storage pools as implemented on the CLARiiON and VNX arrays.
  • Erik Smith has a couple of great FCoE-focused blog posts, first on directly connecting to an FCoE target and then on VE_Ports and multihop FCoE. Both of these posts are in-depth technical articles that are, in my opinion, well worth reading.
  • Brian Norris posted about some limitations with certain FLARE 30 features when used in conjunction with Celerra (DART 6.0). I know that at least one of these limitations—the support for FAST VP on LUNs used by Celerra—are addressed in the VNX arrays.
  • Brian also recently posted some good information on a potential login issue with Unisphere; this is caused by SSL certificates that are generated with future dates.
  • J Metz of Cisco also has a couple of great FCoE-focused posts. In To Tell the Truth: Multihop FCoE, J covers in great detail the various topology options and the differences in each topology. Then, in his post on director-class multihop FCoE, J discusses the products that implement multihop FCoE for Cisco.
  • If you’ve never used EMC’s VSI (Virtual Storage Integrator) plug-in for vCenter Server, have a look at Mike Laverick’s write-up.

OK, that does it for the Storage Edition of Technology Short Take #12. Check back in a couple of days for the Networking Edition of Technology Short Take 12.

Tags: , , , ,

In this post, I want to pull together all the steps necessary to take a Converged Network Adapter (CNA)-equipped server and connect it, using FCoE, to a Fibre Channel-attached storage array. There isn’t a whole lot of “net new” information in this post, but rather I’m summarizing previous posts, organizing the information, and showing how these steps relate to each other. I hope that this helps someone understand the “big picture” of how FCoE and Fibre Channel relate to each other and how they interoperate (which, quite frankly, is one of the key factors for the adoption of FCoE).

The steps involved come from an environment with the following components:

  • A Dell PowerEdge R610 server running VMware ESXi and containing an Emulex CNA
  • A Cisco Nexus 5010 switch running NX-OS 4.2(1)N1(1).
  • A Cisco MDS 9134 Fibre Channel switch running NX-OS 5.0(1a).
  • An older EMC CX3-based array with Fibre Channel ports in the storage processors.

We’ll start at the edge (the host) and work our way to the storage. All these steps assume that you’ve already taken care of the physical cabling.

  1. Depending upon how old the software is on your hosts, you might need to install updated drivers for your CNA, as I described here. If you’re using newer versions of software, the drivers will most likely work just fine out of the box.
  2. The closest piece to the edge is the FCoE configuration on the Nexus 5010 switch. Here’s how to setup FCoE on a Nexus 5000. Be sure that you map VLANs correctly to VSANs; for every VSAN that needs to be accessible from the FCoE-attached hosts, you’ll need a matching VLAN. Further, as pointed out here, the VLAN and VLAN trunking configuration is critical to making FCoE work properly anyway.
  3. The next step is connecting the Nexus 5010 to the MDS 9134 Fibre Channel switch. Read this to see how to configure NPV on a Nexus 5000 if you are going to use NPV mode (and read this for more information on NPV and NPIV). Using NPV or not, you’ll also need to setup connections between the Nexus and the MDS; here’s how to setup SAN port channels between a Cisco Nexus and a Cisco MDS.
  4. Once the Nexus and the MDS are connected, then you’ll need to perform the necessary zoning so that the hosts can see the storage. Before starting the zoning, you might find it helpful to set up device aliases. After your device aliases are defined, you can create the zones and zonesets. This post has information on how to create zones via CLI; this post has information on how to manage zones via CLI.

At this point—if everything is working correctly—then you are done and you should be ready to present storage to the end hosts.

I hope this helps put the steps involved (all of which I’ve already written about) in the right order and in the right relationship to each other. If there are any questions, clarifications, or suggestions, please feel free to speak up in the comments.

Tags: , , , , , , ,

Now that I’ve managed to get my VPLEX clusters up and running, I’m ready to start providing some posts on VPLEX and how it’s configured and used. To get things started, I want to first provide an overview of VPLEX storage objects and how they relate to each other.

Storage volumes: Storage volumes are the LUNs that are presented to the VPLEX back-end ports from the storage array. The screenshot below, taken from the VPLEX management GUI, shows a series of storage volumes taken from two separate back-end arrays.

Extents: Extents are created on storage volumes. You have the option of creating multiple extents on a single storage volume, but EMC generally recommends creating one extent per storage volume. (Future posts will discuss why this is beneficial.)

Devices: Devices are created from extents. You can combine multiple extents together to form a single device, or you can present a single extent up as a single device. As you can see in the screenshot, devices also have a geometry associated with them, like RAID-0 (no mirroring of devices together), RAID-1 (mirroring of devices), or RAID-C (concatenating devices).

Distributed devices: Distributed devices are mirrored devices that are spread across two VPLEX clusters connected together into a metro-plex. Distributed devices require extents from both VPLEX clusters in the metro-plex. Like “regular” devices, distributed devices also have a geometry. As far as I know, all distributed devices will automatically have a RAID-1 geometry. Note that because distributed devices aren’t confined to a single cluster, they reside in a different place in the hierarchy.

Virtual volume: Virtual volumes are built from devices. When a virtual volume is built from a device, it’s referred to just as a virtual volume. When the virtual volume is built from a distributed device, it’s referred to as a distributed virtual volume. Virtual volumes are what are exposed (exported in VPLEX parlance) to hosts via the VPLEX front-end ports.

In future VPLEX posts, I’ll provide information on managing these various storage objects (creating, renaming, and deleting objects) as well as information on integrating VPLEX with other data protection solutions. Stay tuned!

Tags: , , , ,

Last year, I posted a couple of articles on configuring and managing Cisco MDS Fibre Channel zones via the CLI:

New User’s Guide to Configuring Cisco MDS Zones via CLI
New User’s Guide to Managing Cisco MDS Zones via CLI

In those posts, I discussed the use of the fcalias command to create aliases for World Wide Port Names (WWPNs) on the fabric. A couple of people suggested via Twitter and blog comments that I should use device aliases instead of the the fcalias command. As a follow up to those posts, here is some information on using device aliases on a Cisco MDS Fibre Channel switch.

To create a device alias, you’ll use the device-alias database command in global configuration mode. Once you are in database configuration mode, you can create device aliases using the device-alias command, like this:

mds(config)# device-alias database
mds(config-device-alias-db)# device-alias name <Friendly name> pwwn <Fibre Channel WWPN>
mds(config-device-alias-db)# exit
mds(config)# end

There is an additional step required after defining the device aliases. You must also commit the changes to the device alias database, like this:

mds(config)# device-alias commit

This commits the changes to the device alias database and makes the device aliases active in the switch.

Once a device alias is created, it applies to that WWPN regardless of VSAN. This means that you only have to define a single device alias for any given WWPN, whereas with the fcalias command a different alias needed to be defined for each VSAN. All other things being equal (and they’re not equal, as you’ll see in a moment), that alone is worth switching to device aliases in my opinion.

Using device aliases also provides a couple other key benefits:

  • Device aliases are automatically distributed to other Cisco-attached switches. For example, I defined the device aliases on a Cisco MDS 9134 that was attached to the Fibre Channel expansion port of a Cisco Nexus 5010 switch. The Nexus switch automatically picked up the device aliases. As best I can tell, this is controlled by the device-alias distribute global configuration command (or its reverse, the no device-alias distribute, which would disable device alias distribution).
  • Once a device alias is defined for a WWPN, anytime the WWPN is displayed the device alias is also displayed. So in the output of various commands like show flogi database, show fcns database, or show zone you will see not only the WWPN, but also that WWPN’s associated device alias.
  • You can use the device alias as the destination with the fcping command.

All in all, I see a lot of value in using device aliases over simple Fibre Channel aliases. I’ll grant that some of this value is more readily apparent only in homogenous Cisco storage networks, but even in single-switch networks I personally would use device aliases.

To those who suggested I look at device aliases, I thank you! You’ve made my job easier.

As always, I welcome your feedback! Feel free to speak up in the comments with corrections, clarifications, or suggestions.

Tags: , , ,

As part of an ongoing effort to expand the functionality of the vSpecialist lab in the EMC RTP facility, we recently added a pair of Cisco MDS 9134 Fibre Channel switches. These Fibre Channel switches are connected to a pair of Cisco Nexus 5010 switches, which handle Unified Fabric connections from a collection of CNA-equipped servers. To connect the Nexus switches to the MDS switches, we used SAN port channels to bond multiple Fibre Channel interfaces together for both redundancy and increased aggregate throughput. Here is how to configure SAN port channels to connect a Cisco Nexus switch to a Cisco MDS switch.

If you are interested, more in-depth information can be found here on Cisco’s web site.

Although I’ve broken out the configuration for the MDS and the Nexus into separate sections, the commands are very similar. In my situation, the MDS 9134 was running NX-OS 5.0(1a) and the Nexus 5010 was running NX-OS 4.2(1)N1(1).

Configuring the Cisco MDS 9134

To configure the MDS 9134 with a SAN port channel, use the following commands.

First, create the SAN port channel with the interface port-channel command, like this:

mds(config)# interface port-channel 1

You can replace the “1″ at the end of that command with any number from 1 to 256; it’s just the numeric identifier for that SAN port channel. The SAN port channel number does not have to match on both ends.

Once you’ve created the SAN port channel, then add individual interfaces with the channel-group command:

mds(config)# interface fc1/16
mds(config-if)# channel-group 1

The “1″ specified in the channel-group command has to match the number specified in the earlier interface port-channel command. This might seem obvious, but I wanted to point it out nevertheless.

Repeat this process for each interface you want to add to the SAN port channel. In my example, I used two interfaces.

When you add an interface to the SAN port channel, NX-OS reminds you to perform a matching configuration on the switch at the other end, then use the no shutdown command to make the interfaces (and the SAN port channel) active. Let’s look first at the commands for configuring the Nexus, then we’ll examine what it looks like when we bring the SAN port channel online.

Configuring the Cisco Nexus 5010

The commands here are very similar to the MDS 9134. First, you need to create the SAN port channel using the interface san-port-channel command (note the slight difference in commands between the MDS and the Nexus here):

nexus(config)# interface san-port-channel 1

As with the MDS, the number at the end simply serves as a unique identifier for the SAN port channel and can range from 1 to 256.

Then add interfaces to the SAN port channel using the channel-group command:

nexus(config)# interface fc2/1
nexus(config-if)# channel-group 1
nexus(config-if)# interface fc2/2
nexus(config-if)# channel-group 1

As I’ve shown above, simply repeat the process for each interface you want to add to the SAN port channel. As on the MDS, NX-OS reminds you to perform a matching configuration on the opposite end of the link and then issue the no shutdown command.

Bringing Up the SAN Port Channel

Once a matching configuration is performed on both ends, then you can use the no shutdown command (which you can abbreviate to simply no shut) to activate the interfaces and the SAN port channel. After activating the interfaces, a show interface port-channel (on the MDS) or a show interface san-port-channel (on the Nexus) will show you the status of the SAN port channel. Only the first few lines of output are shown below (this output is taken from the Nexus):

nexus# sh int san-port-channel 1
san-port-channel 1 is trunking (Not all VSANs UP on the trunk)
    Hardware is Fibre Channel
    Port WWN is 24:01:00:05:9b:7b:0c:80
    Admin port mode is auto, trunk mode is on
    snmp link state traps are enabled
    Port mode is TE
    Port vsan is 1
    Speed is 4 Gbps
    Trunk vsans (admin allowed and active)  (1)
    Trunk vsans (up)                        ()
    Trunk vsans (isolated)                  ()
    Trunk vsans (initializing)              (1)

A couple of useful pieces of information are available here:

  • First, you can see that the SAN port channel is not fully up; it’s still initializing. This is shown by the “Not all VSANs UP on the trunk” message, as well as by the “Trunk vsans (initializing)” line.
  • Second, you can see the only a single member is up. Note the speed of the SAN port channel is listed as 4 Gbps.
  • Third, note that this is a trunking port, meaning that it could carry multiple VSANs. This is noted by the “Port mode is TE” line as well as the first line of the output (“san-port-channel 1 is trunking”).

As it turns out, I’d cabled the connections wrong; after I fixed the connections and gave the SAN port channel a small amount of time to initialize, the output was different (this output is taken from the MDS):

nexus# sh int port-channel 1
port-channel 1 is trunking
    Hardware is Fibre Channel
    Port WWN is 24:01:00:05:73:a7:72:00
    Admin port mode is auto, trunk mode is on
    snmp link state traps are enabled
    Port mode is TE
    Port vsan is 1
    Speed is 8 Gbps
    Trunk vsans (admin allowed and active)  (1)
    Trunk vsans (up)                        (1)
    Trunk vsans (isolated)                  ()
    Trunk vsans (initializing)              ()

Now you can see that both members of the SAN port channel are active (“Speed is 8 Gbps”) and that all VSANs are trunking across the SAN port channel.

At this point, you are now ready to proceed with creating VSANs, zones, and zonesets. Refer to these articles for more information on MDS zone creation and management via CLI:

New User’s Guide to Configuring Cisco MDS Zones via CLI
New User’s Guide to Managing Cisco MDS Zones via CLI

As always, questions, clarifications, or corrections are welcome—just add them below in the comments. Thanks!

Tags: , , , ,

Storage Short Take #5

I’ve decided to resurrect my Storage Short Take series, after almost a year since the last one was published. I find myself spending more and more time in the storage realm—which is completely fine with me—and so more and more information coming to me in various forms is related to storage. While I’m far from the likes of storage rockstars such as Robin Harris, Stephen Foskett, Storagebod, and others, hopefully you’ll find something interesting and useful here. Enjoy!

  • This blog post by Frank Denneman on the HP LeftHand product is outstanding. I learned more from this post than a lot of posts recently. Great work Frank!
  • Need a bit more information on FCoE? Nigel Poulton has a great post here (it’s a tad bit older, but I’ve just stumbled across it) with good details for those who might not be familiar with FCoE. It’s worth a read if you haven’t already taken the time to come up to speed on FCoE and its “related” technologies.
  • What led me to Nigel’s FCoE post was this post by Storagezilla in which he rants about “vendor flapheads” who “are intentionally obscuring it’s [FCoE's] limitations”. You’ve got that right! Wanting to present a reasonably impartial and complete view of FCoE was partially the impetus behind my end-to-end FCoE post and the subsequent clarification. Thankfully, I think that the misinformation around FCoE is starting to die down.
  • This post has a bit of useful information on HP EVA path policies and vSphere multipathing. I would have liked a bit more detail than what was provided, but the content is good nevertheless.
  • Devang Panchigar’s recoup of HP TechDay day 1, which focused on HP StorageWorks technologies, has some good information, especially if you aren’t already familiar with some of HP’s various storage platforms.
  • Chad Sakac of EMC has some very useful information on Asymmetric Logical Unit Access (ALUA), VMware vSphere, and EMC CLARiiON arrays. If you using EMC storage with your VMware vSphere 4 environment, and you have a CX4, and you’re running FLARE 28.5 or later, it might be worthwhile to switch your path policy from NMP to Round Robin (RR).
  • Speaking of RR with vSphere, somewhere I remember seeing information on changing the default number of I/Os down a path, and tweaking that for best performance. Was that in Chad’s VMworld session? Anyone remember?
  • If you’re looking for a high-level overview of SAN and NAS virtualization, this InfoWorld article can help you get started. You’ll soon want to delve deeper than this article can provide, but it’s a reasonable starting point, at least.

That’s it for this time around. Feel free to share other interesting or useful links in the comments.

Tags: , , , , , ,

By Aaron Delp
Twitter: aarondelp
FriendFeed (Delicious, Twitter, & all my blogs in one spot): aarondelp

This week I’ve had the privilege of attending a Cisco Nexus 5000/7000 class. I have learned a tremendous amount about FCoE this week and after some conversations with Scott about the topic, I wanted to tackle it one more time from a different point of view. I have included a list of some of Scott’s FCoE articles at the bottom for those interested in a more in-depth analysis.

Disclaimer: I am by no means an FCoE expert! My working knowledge of FCoE is about four days old at this point. If I am incorrect in some of my situations below, please let me know (keep it nice and professional, people!) and I will be happy to make adjustments.

If you are an existing VMWare customer today with FC as your storage transport layer, should you be thinking about FCoE? How would you get started? What investments can you make in the near future to prepare you for the next generation?

These are questions I am starting to get from my customers in planning/white board sessions around VMware vSphere and the next generation of virtualization. The upgrade to vSphere is starting to prompt planning and discussions around the storage infrastructure.

Before I tackle the design aspect, let me start out with some hardware and definitions.

Cisco Nexus 5000 series switch: The Nexus 5K is a Layer 2 switch that is capable of FCoE and can provide both Ethernet and FC ports (with an expansion module). In addition to Ethernet switching, the switch also operates as an FC fabric switch providing full fabric services, or it can be set for N_Port Virtualization (NPV) mode. The Nexus 5K can’t be put in FC switch mode and NPV mode at the same time. You must pick one or the other.

N_Port Virtualization (NPV) mode: NPV allows the Nexus 5K to act as a FC “pass thru” or proxy. NPV is great for environments where the existing fabric is not Cisco and merging the fabrics could be ugly. There is a downside to this. In NPV mode, no targets (storage arrays) can be hung off the Nexus 5K. This is true for both FC and FCoE targets.

Converged Network Adapter (CNA): A CNA is single PCI card that contains both FC and Ethernet logic, negating the need for separate cards, separate switches, etc.

Now that the definitions and terminology is out of the way, I see four possible paths if you have FC in your environment today.

1. FCoE with a Nexus 5000 in a non-Cisco MDS environment (merging)

In this scenario, the easiest way to get the Nexus on the existing non-Cisco FC fabric is to put the switch in NPV mode. You could put the switch in interop mode (and all the existing FC switches), but it is a nightmare to get them all talking and you often lose vendor specific features in interop mode. Plus, to configure interop mode, the entire fabric has to be brought down. (You do have redundant fabrics, right?)

With the Nexus in NPV mode, what will it do? Not much. You can’t hang storage off of it. You aren’t taking advantage of Cisco VSANs or any other features that Cisco can provide. You are merely a pass thru. The zoning is handled by your existing switches; your storage is off the existing switches, etc.

Why would you do this? By doing this, you could put CNAs in new servers (leaving the existing servers alone) to talk to the Nexus. This will simplify the server side infrastructure because you will have fewer cables, cards, switch ports, etc. Does the cost of the CNA and new infrastructure offset the cost of just continuing the old environment? That is for you to decide.

2. FCoE with a Nexus 5000 in a non-Cisco MDS environment (non-merging)

Who says you have to put the Nexus into the existing FC fabric? We have many customers that purchase “data centers in a box”. By that I mean a few servers, FC and/or Ethernet switches, storage, and VMware all in one solution. This “box” sits in the data center and the network is merged with the legacy network, but we stand up a Cisco SAN next to the existing non-Cisco SAN and just not let them talk to each other. In this instance, we would use CNAs in the servers, Nexus as the switch, and you pick a storage vendor. This will work just like option 3.

3. FCoE with a Nexus 5000 in a Cisco MDS environment

Now we’re talking. Install the Nexus in FC switch mode, merge it with the MDS fabric, put CNAs in all the servers and install the storage off the Nexus as either FC or FCoE. You’re off to the races!

You potentially gain the same server side savings by replacing FC and Ethernet in new servers with CNAs. You are able to use all of the Cisco sexy features of FCoE. Nice solution if the cost is justified in your environment.

4. Keep the existing environment and use NFS to new servers

What did I just say? Why would I even consider that option?

OK, this last one is a little tongue-in-cheek for customers that are already using FC. The NFS vs. traditional storage for VMWare is a bit of a religious debate. I know you aren’t going to sway me and I know I’m not going to sway you.

I admit I’m thinking NetApp here in a VMWare environment; I’m a big fan so this is a biased opinion. NetApp is my background but other vendors play in this space as well. I bet Chad will be happy to leave a comment to help tell us why (and I hope he does!).

Think of it this way. You’re already moving from FC cards to CNAs. Why not buy regular 10Gb Ethernet cards instead? Why not just use the Nexus 5K as a line-rate, non-blocking 10Gb Ethernet switch? This configuration is very simple compared to FCoE at the Nexus level and management of the NetApp is very easy! Besides, you could always turn up FCoE on the Nexus (and the NetApp) at a future date.

In closing, I really like FCoE but as you can see it isn’t a perfect fit today for all environments. I really see this taking off in 1-2 years and I can’t wait. Until then, use caution and ask all the right questions!

If you are interested in some more in-depth discussion, here are links to some of Scott’s articles on FCoE:

Continuing the FCoE Discussion
Why No Multi-Hop FCoE?
There Might Be an FCoE End to End Solution

Tags: , , , , , , , ,

Here’s Virtualization Short Take #27, a collection of news, tidbits, thoughts, articles, and useless trivia I’ve gathered over the last week or so. Perhaps you’ll find a diamond in the rough among these items!

  • Interested in more information on how it is, exactly, that Cisco is going to provide so much memory in their UCS blades and rack mount servers to make them ideal virtualization hosts? This article from CommsDesign and this “Catalina” article by Rodos Haywood both provide some great information on how Cisco is working around the Intel reference architecture limitations introduced with the Xeon 5500 and Quick Path Interconnect (QPI).
  • This article provides a handy reference on how to unregister the Nexus 1000V vCenter Server plug-in. I wish I’d known this information several weeks ago…
  • Need to view some configuration files on an ESX host? Just browse to http://<IP address of ESX server>/host and you’re all set. I learned of this handy little trick via Virtual Foundry.
  • And speaking of handy little tips, here’s one Eric Sloof shared regarding the vCenter Ops Dashboard. Again, just browse over to http://<IP address of vCenter Server>/vod/index.html to view the vCenter Ops Dashboard.
  • Adam Leventhal describes using the latest version of VirtualBox—which now includes OVF support and host-only networking—to run the Sun Storage 7000 Simulator. This is pretty cool stuff. I hope Oracle doesn’t kill it like Virtual Iron…
  • I just mentioned Virtual Foundry a bit ago, but forgot to mention this great post on hardening the VMX file. Good stuff.
  • For others who are, like myself, pursuing the elusive VMware Certified Design Expert (VCDX) certification, Duncan’s recent post describing the VCDX design defense is a great resource. Thanks, Duncan!
  • The VMware networking team addresses some questions around using VMware for virtualized DMZs, and how to protect against Layer 2 attacks.
  • Want to do manual linked clones in VMware Fusion? Here’s how.
  • Via Matt Hensley, I found this VIOPS document on configuring a VMware vCenter Data Recovery dedupe store.
  • This article has more information on installing ESXi 4.0 to a flash drive, a process I have yet to try. (Instructions for burning ESXi 3.5 to a flash drive can be found here.) Anyone else done it yet? I’d be interested in how it went.
  • If you have any questions about SAN multipathing, Brent Ozar’s two part series on the topic may help straighten things out (here’s Part 1 and Part 2). I’m not sure that I agree with Brent’s statement regarding the ability of desktop-class SATA drives to saturate 4Gbps Fibre Channel, but I’m no storage expert so I could very well be wrong.
  • VMware SE and friend Aaron Sweemer provides a handy script that can help fix Service Console networking when performing automated builds of VMware ESX.

That wraps it up for this edition of Virtualization Short Takes. Feel free to share thoughts, questions, or corrections in the comments, and thanks for reading!

Tags: , , , , , ,

« Older entries