Nexus

You are currently browsing articles tagged Nexus.

Some time ago I posted a “how to” article on how to configure FCoE on a Nexus 5000 switch. At that time, I did not put the Nexus 5000 into NPV mode but rather connected it to a Cisco MDS Fibre Channel switch without using NPV. In this entry, I’d like to follow up on that article and show you how to configure NPV on a Nexus 5000.

If you aren’t familiar with NPV (N_Port Virtualization) and how it’s different than NPIV (N_Port ID Virtualization), check out this article titled “Understanding NPIV and NPV”; It should help clear things up. Because NPV makes the Nexus 5000 look like an NPIV-enabled host, one potential use for NPV, as in this case, is when connecting the Nexus 5000 to a non-Cisco Fibre Channel switch. Using NPV helps ease interoperability concerns. In this instance, I’m connecting the Nexus 5000 to a Brocade Fibre Channel switch (actually an EMC Connectrix).

Note that I tested these instructions on a Nexus 5010 using both NX-OS 4.1(3)N2(1) as well as NX-OS 4.2(1)N1(1).

The first step is to enable NPV on the Nexus 5000. As far as I can tell, in order to enable NPV you must first enable FCoE using the feature command:

switch(config)# feature fcoe

This loads various Fibre Channel modules and makes possible other features, including NPV and NPIV. Enabling NPV erases the switch configuration and reboots the switch, so be sure you are connected via a console connection before enabling NPV with the feature command:

switch(config)# feature npv

Immediately after enabling NPV, the Nexus 5000 will reboot (you’re warned and given the option to proceed or cancel). The warning indicates that the switch’s configuration will be removed, but the minimally-configured switches I used in my testing retained their configuration. Granted, I hadn’t performed any Fibre Channel or FCoE configurations yet, so perhaps that’s the configuration to which the warning was referring.

Once NPV is enabled on the switch, you can then configure Fibre Channel uplink ports as NP ports (proxy N_ports); these are also referred to as external interfaces. To configure a Fibre Channel port as an NP port, use these commands:

switch# config t
switch(config)# interface fc slot/port
switch(config-if)# switchport mode np
switch(config-if)# no shut
switch(config-if)# exit
switch(config)# exit

You should then be ready to physically connect to the upstream Fibre Channel switch, which—if you recall correctly from my earlier NPV/NPIV post—needs to be NPIV-enabled. In this particular case, I was uplinking the Nexus 5010 to an EMC-rebranded Brocade switch (a Connectrix DS-300B running Fabric OS 6.1.0). To show whether the port on the Connectrix was enabled for NPIV, I used the portcfgshow command:

rtp-fc-sw-01:admin> portcfgshow port number

Look for the line that says “NPIV Capability”; the value should be reported as “ON”. If the value is not “ON”, you’ll need to use the portcfgnpivport command to enable NPIV on the specified port, like this:

rtp-fc-sw-01:admin> portcfgnpivport port number 1

The “1″ at the end of that command enables NPIV; a “0″ would disable NPIV for that port.

Once NPIV is enabled on the upstream Fibre Channel switch, when you physically connect the configured external interface then the Fibre Channel link should come up. I used the show int fc slot/number command on the Nexus to verify that the port was up; on the Connectrix, I used the portshow port command. In addition, I was also able to see the Nexus switch logged into the Fibre Channel fabric on the Connectrix using the nsshow command.

Once you have Fibre Channel connectivity via the external interfaces, then configuring FCoE to hosts connected to the Nexus follows the same set of instructions laid out in my earlier FCoE how-to article.

From that point on, it’s only a matter of configuring zones (see here for help with zones on a Cisco MDS) and presenting storage. Those are different posts for a different day…

Tags: , , , , , ,

Welcome to the 40th post in the Virtualization Short Take series, where I share with you various virtualization-related links, thoughts, and news tidbits. (Occasionally, I throw in some stuff that’s not virtualization related just to see if you are paying attention.) Enjoy!

  • There have been a couple of posts now discussing Storage IO Control, a new feature that is possibly slated for inclusion in a future release of VMware vSphere. Storage IO Control extends the disk shares model cluster-wide, allowing administrators to properly shape access to back-end storage resources. The inimitable Scott Drummonds discussed it on Pivot Point (his blog), and Craig Stewart also recently published an article about Storage IO Control over at Gestalt IT. There’s a fair amount of duplication between the two articles (Craig based his article partly on Scott’s), but both are worth a read if you need to come up to speed on this new feature. (Quick disclaimer: I’m discussing Storage IO Control here only because it’s been mentioned elsewhere by others. As to whether or not this feature will or will not appear in a future release, or when that future release might be, I know nothing. OK?)
  • I don’t know why, but I saw this virtual appliance on VMware’s web site earlier today and it has triggered a nagging feeling in the back of my head. Is the future of computing found in simple “scale out” building blocks like this?
  • This article on “VM stall” by Andi Mann just re-confirms something I’ve been saying for a while: there are still too many companies out there that aren’t taking full advantage of virtualization. If you’re one of the almost 54% of companies that is still less than 30% virtualized, what’s holding you back?
  • Among all the other announcements from EMC World last week, this little tidbit might have gotten overlooked. Chad blogged about a fix contained in FLARE 30—the updated version announced at the conference—that addresses a problem with iSCSI initiators and the CLARiiON arrays. Good work, EMC engineering!
  • Over on VMware’s ESXi Chronicles blog, Charu Chaubal recently published a two-part series on hardware monitoring via CIM (part 1 and part 2).
  • A reader dropped me an e-mail about an issue uncovered in their environment while trying to automate the VMware Tools installation. Apparently, the VMware Tools installation relies on 8.3 file naming conventions. Normally, this wouldn’t be a problem, but in environments where 8.3 file name creation is disabled…well, you can see where there might be a problem. No workaround has yet been found. Any wizards out there who have suggestions are welcome to add them to the comments of this post.
  • Two posts popped up in the last couple of weeks regarding the default number of ports on a Nexus 1000V port profile: this post by Kevin Goodman and this post by Jason Nash. Fortunately, it’s a quick process to increase the default maximum of 32 ports.
  • Running your VMware vSphere environment on NFS? Have a look at this document from VMware.
  • Didier Pironet of DeinosCloud, the same gentleman who showed us how to increase the number of VMware HA primary nodes, posted a guide on adjusting the memory usage of Tomcat (the engine behind VMware vCenter’s web services). This is most likely an unsupported configuration change, but it might be handy in test/development environments.
  • Kevin Goodman also had a good post on configuring EMC PowerPath on Linux on Cisco UCS. I know, I know: this isn’t strictly related to virtualization, but it’s close enough in my book.
  • Alastair of demitasse.co.nz finally declares that the emperor has no clothes when he states why he believes users don’t want a client hypervisor. Personally, I tend to agree with him; I think a hosted hypervisor is far more valuable on the client-side space (especially in the BYOPC scenario). Just because you can run a bare metal hypervisor on your laptop doesn’t necessarily mean that you should run a bare metal hypervisor on your laptop.
  • There seems to be a fair amount of confusion around the vStorage APIs; perhaps this is due to the different subsets of the vStorage APIs. There are the vStorage APIs for Array Integration (VAAI); these were discussed in some detail last week at EMC World. There are also the vStorage APIs for Multipathing (VAMP), which serve to support multipathing plugins like PowerPath/VE. Finally, there are the vStorage APIs for Data Protection (VADP), which are the APIs that serve to replace VMware Consolidated Backup. If you’d like to know more about VADP in particular, this VMware KB article has a list of frequently asked questions about VADP.
  • Tom Howarth has brought to light a potentially serious problem with Changed Block Tracking (CBT), a key part of the vStorage APIs for Data Protection (VADP) that enables lots of backup and recovery applications.
  • While reviewing one of the weekly VMware KB digests, I came across this VMware KB article in which virtual NICs are sometimes detected as removable hardware; this can, in turn, cause the virtual NIC to disappear from the virtual machine. It appears that the only workaround for this behavior is to disable HotPlug.
  • Xsigo recently posted a comparison of their I/O virtualization solution vs. other I/O virtualization solutions. They include FCoE as an I/O virtualization solution, but as I’ve said in the past I don’t consider FCoE an I/O virtualization solution. To include FCoE in this sort of comparison is kind of like saying that an apple is a poor orange because it lacks a thick outer skin. FCoE wasn’t designed to do I/O virtualization—it was designed to carry Fibre Channel traffic over Ethernet. Despite the liberty with which comparative technologies are selected, the article is worth reading nevertheless.

And to round out this issue of Virtualization Short Takes, here are a few “bonus links” I found:

UCS with disjointed L2 domains
The “Mini-Rack” Approach to Blade Server Design
Hot adding or removing a Cisco 3750 from a stack
EMC World Cubed – Here’s all the Video
ESXTOP, My understanding

That’s it for now. I hope you found something useful. Feel free to share more useful links in the comments, and thanks for reading!

Tags: , , , , , , ,

The announcement of VPLEX last week at EMC World 2010 in Boston has introduced lots of new possibilities in how virtualized data centers should be designed and deployed. Of particular interest to a number of people is how VPLEX interacts with VMware Site Recovery Manager (SRM).

Jason Nash of Varrow mulls over VPLEX and VMware SRM in this post, where he asks if VMware SRM is even necessary:

If money was no object, or if you needed VPLEX anyway, where does that leave VMware’s Site Recovery Manager?

While many of Jason’s points are absolutely valid, there are some other very important things to keep in mind:

  1. First, and perhaps most importantly, VPLEX and VMware SRM is not an “OR” discussion, it’s an “AND” discussion. As noted by Chad Sakac’s response, there is a VPLEX SRA currently planned. There are some issues to work through—such as the current requirement by VMware SRM for two vCenter Server instances—but I’m confident these issues will be resolved.
  2. Second, the behavior of VPLEX in the event of an unplanned outage lends itself well to VMware SRM-like behavior. In the event that the VPLEX cluster in Site A (your primary site) loses connectivity to the VPLEX cluster in Site B (your secondary/DR/failover site), a set of rules defined by the user control which cluster will continue to have read/write access to distributed devices. The other cluster will suspend I/O to distributed devices until a user manually resumes I/O. This makes VPLEX behavior in an unplanned outage act a lot like VMware SRM already, and is probably one of the reasons why a VPLEX SRA is under development. As long as there are steps that can be automated in some programmatic way, there is continued value for VMware SRM.
  3. Third, it’s important to keep in mind that the requirements for vMotion over distance include spanned Layer 2 VLANs (using something like Overlay Transport Virtualization from Cisco); VMware SRM has no such requirement. Further, VPLEX is currently limited to synchronous distances; VMware SRM is limited only by the underlying replication mechanisms. This means that VMware SRM continues to be a very valid deployment option even in organizations that may also deploy VPLEX.

Jason does make one point that I absolutely agree with:

But, to me, this yet again adds more reasons to virtualize all the applications that you can. Virtualizing the tier 1 apps reduces your DR plan by a large amount with SRM and this would make it even simpler.

The more applications you virtualize, the more you will be able to take advantage of products like VMware SRM and VPLEX. So what are you waiting for? Get virtualizing!

Tags: , , , , , ,

It’s been a while since I last posted a “Thinking Out Loud” post, but a Twitter discussion this morning prompted me to post this one. Last night, I posted a question to Brad Hedlund (@bradhedlund on Twitter and a great data center networking resource) regarding the Nexus 2232PP fabric extender. My tweet to Brad was this: does the Nexus 2232PP make “multi-hop FCoE” possible via NIV?

(If you haven’t already read my post on the relationship between FCoE and NIV, go read it now as it provides a good background and context for this discussion.)

Brad responded, confirming my assessment and stating that FIP snooping wasn’t necessary. I wasn’t sure about the FIP snooping part but after seeing the response I now understand. In any event, a number of others starting questioning my use of “multi-hop FCoE” in conjunction with the 2232PP fabric extender, stating that it wasn’t a hop because ti does not actually switch any traffic.

Strictly speaking, all of these individuals are absolutely correct; for more information, go read this post on NIV. In this case, the Nexus 5000 is the interface virtualization (IV)-capable bridge and the Nexus 2232PP is the interface virtualizer (IV). The IV doesn’t switch any traffic; all switching occurs on the IV-capable bridge. Therefore, from a switching perspective, a Nexus 5000 and any or all associated fabric extenders are a single hop.

My response is this: what is multi-hop, anyway? Is it the presence of multiple switches in a data path between an initiator and a target? Or is it the presence of multiple physical devices, chained together, between an initiator and a target? In the first definition, using a fabric extender isn’t multi-hop; in the second definition, it is. More to the point, should a customer really care which definition is correct? Why is multi-hop FCoE important, anyway? I would say the answer to that question is scalability. Customers want to be able to scale the FCoE fabrics larger than they can today. Multi-hop FCoE—however you want to define it—makes that possible. So does it really matter how we get there? Besides which point, using the fabric extender approach means only a single point of management instead of multiple points of management. Isn’t that better?

What do you think? Do your own “thinking out loud” in the comments below.

Tags: , , , ,

It’s that time again: time for another Virtualization Short Take! My Virtualization Short Takes are quick glances at various news bytes, announcements, useful blog posts, or other items of interest. (By the way, the “short” in “Short Take” does not imply that my post is going to be short, in case anyone was wondering. I’m still long-winded, and I have a lot of things that I find interesting.)

  • Have I mentioned how useful the weekly VMware KB digests are?
  • Frank Denneman has published a couple of really great articles recently. The first discussed how to remove an orphaned Nexus 1000V distributed virtual switch; the second discusses a complex interaction between HP Continuous Access and LUN balancing scripts. Both articles are worth a read.
  • Similarly, Jeremy Waldrop has had a couple of good posts since he managed to get his hands on a Cisco UCS. The first post describes a “Doh!” moment when Jeremy realizes that adding more vNICs to a VMware ESXi instance with the Cisco VIC (aka “Palo”; sorry, Cisco, you’re not going to be escaping that code name any time soon) is really just a matter of specifying them in the service profile. I can certainly see where that’s not immediately intuitive. The other article describes Jeremy’s experience with using vNIC failover. There’s great information in the comments to that article; in particular, be sure not to enable vNIC failover with VMware vSphere. Bad things happen as a result. (OK, maybe not “bad things,” but network connectivity can be adversely affected. You should let VMware vSphere handle the NIC teaming and failover.)
  • Toni Westbrook has a good article on how to move the COS VMDK in VMware ESX 4.0. Key note: this solution is currently unsupported by VMware, so use at your own risk.
  • I’ve mentioned before how various bloggers often have a “masterpiece” post. This isn’t necessarily their most well-written post, but it’s the post that, for whatever reason, is a defining post for them. For me, it’s the ESX/VLANs/NIC teaming article I wrote in 2006. I think Jason Boche might have just come up with his: an in-depth discussion of the vpxd.cfg configuration options. Great information, Jason!
  • In VDI environments, storage capacity is only one aspect of the overall storage equation. Vijay Swami at vEverything takes a pretty balanced view of how two leading storage vendors—EMC and NetApp—address not only storage capacity, but also IOPS. It’s worth a read and again underscores that there is no “one right way” to do things. Different doesn’t necessarily mean better or worse, just different. It’s all about the technology choices. (Disclosure: I work for EMC Corporation.)
  • VDI on local disks, anyone? It’s an interesting discussion point that has its pros and cons. I guess the value of this sort of design really depends upon the business objectives the VDI implementation is trying to fulfill.
  • Is anyone else amused by the abrupt “about face” that Microsoft performed with Hyper-V’s dynamic memory feature? Wow…even I was caught off-guard by how quickly they went from one end of the spectrum to the other. I would rather hear someone say, “You know, we were wrong, and this is a valuable feature after all” than to just flip 180 degrees and start moving in a whole new direction.
  • Speaking of Microsoft and whole new directions…there was a great deal of coverage about Microsoft’s desktop virtualization announcement. I won’t try to delve into the details here; that’s a particular niche that is better served by those who have the time to dedicate to it. If you haven’t seen the news, my good friend Alessandro has a great write-up and there’s the official press release from Microsoft.
  • If you’re interested in getting more information on RemoteFX—which appears, more than anything else, to simply be a set of LAN-only acceleration features for RDP and not an entirely new protocol—this article has good information. You might also have a look at this post about Service Pack 1 for Windows Server 2008 R2, which will enable both RemoteFX as well as the afore-mentioned Dynamic Memory.
  • Continuing along with my little BSD love-fest, I came across this article that describes some strange behavior with CARP that can only be fixed by using link aggregation. The geek in me wants to go test this in a bunch of different scenarios to see if the Nexus 1000V fixes it or something like that, but I doubt that I’ll have the time.
  • This is old news now, but in case you hadn’t heard VMware is licensing technology from Likewise Software for use with the next version of VMware vSphere. This will tighten vSphere’s integration with Active Directory. This is generally good, except that it will render my articles on ESX integration with Active Directory useless.
  • With VMware vSphere 4.0 Update 1, you can now install EMC PowerPath/VE using vCenter Update Manager. This VMware KB article provides the details how it’s done.
  • If you’re using ESXi and want to direct logging data elsewhere via syslog, this VMware KB article describes to configure syslog in ESXi.
  • The ages-old discussion of scale up vs. scale out is revisited again in this blog post. I guess the key takeaway for me is the reminder that while VMware HA does restart workloads automatically, there’s still an outage. If you’re running 50 VMs on a host, you’re still going to have an outage across as many as 50 different applications within your organization. That’s not a trivial event. I think a lot of people gloss over that detail. VMware HA helps, but it’s not the ultimate solution to downtime that people sometimes portray it as.
  • PHD Virtual has released esXpress version 4.0 today. I’ve taken a step back from most product announcements simply because they come too quickly to really keep up with them (unless you’re a madman like David Marshall over at VMBlog.com—my hat’s off to you, David!), but the timing worked out for this one. Go have a look at PHD Virtual’s web site for all the details.
  • Last, but most certainly not least, my esteemed colleague Mike Laverick has completed his updated VMware SRM book, now updated for VMware SRM 4.0. Great work, Mike! I would wish you all financial success with the book, but as you’re giving it away for free (an admirable step, by the way) I guess I’ll just have to wish you all other forms of success!

That does it for me this time around, folks. Thanks for reading (I appreciate it!), and if you have some good information to add please feel free to speak up in the comments.

Tags: , , , , , , ,

Welcome to Virtualization Short Take #34, my occasionally-weekly collection of virtualization-related links, posts, and comments. As usual, this information is a hodge-podge of information I’ve gathered from across the Internet over the last few weeks. I hope that you find something useful or helpful here, and thanks for reading!

  • First up is Arne Fokkema’s PowerCLI script to check Windows VM partition alignment. As one commenter pointed out, the fact that the starting offset isn’t 65536—which is what Arne’s script checks—doesn’t necessarily mean that it isn’t aligned. Generally, you can align a Windows partition by setting the starting offset to any number that is evenly divisible by 4096 (4K). If I’m not mistaken, setting the partition offset to 65536 (64K) also ensures that the partition is stripe-aligned on EMC arrays.
  • Here’s a useful reminder to be sure to keep your dependencies in mind when designing VMware vSphere 4 environments. If you design your environment to rely upon DNS—a common situation, since VMware HA is particularly sensitive to name resolution—then be sure to appropriately architect the DNS infrastructure. This “circular dependency” is one reason why I personally tend to keep vCenter Server on a physical system. Otherwise, you have the virtualization management solution running on the infrastructure it is responsible for managing. (Yes, I know that it’s fully supported for it to be virtualized and such.)
  • Forbes Guthrie’s article on incorporating Active Directory authentication and sudo into the kickstart process is a good read. With regard to his note about enabling root SSH access because of an inability to access the Active Directory DCs: I know that in ESX 3.x you could still log in at the Emergency Console when Active Directory connectivity was unavailable; does anyone know if this is still the case with ESX 4.0? I haven’t taken the time to test it yet.
  • Oh, and speaking of Active Directory authentication, Forbes also published this note about Likewise AD authentication supposedly included in ESX 4.1. Looks like someone at Likewise accidentally spilled the beans…
  • I’m sure that everyone has seen the article by Duncan about the ESX 3.x bug that prevents NIC teaming load balancing from working on the global vSwitch configuration, but if you haven’t—well, now you have. Here’s the corresponding KB article, also linked from Duncan’s article. Duncan also recently published a note about an error while installing vCenter Server that is related to permissions; read it here.
  • Are there even better days ahead for virtualization and those involved in virtualization? David Greenfield of Network Computing seems to think so. The comments in the article do seem to bear out my statements that virtualization experts now need to move beyond consolidation and start helping customers tackle the Tier 1, high-end applications. I believe that this is going to require more planning, more expertise, and more knowledge of the applications’ behaviors in order to be successful.
  • Stephen Dion of virtuBLOG brings up a compatibility issue with Intel quad-port Gigabit Ethernet network adapters when used with VMware ESX 4.0 Update 1. Anyone have any updates or additional information on this issue?
  • If you’re considering virtualizing Exchange Server 2010 on VMware vSphere, be sure to read Kenneth’s article here about Exchange 2010 DAGs and VMotion. At least live migration isn’t supported on Hyper-V, either.
  • Want to run a VM inside a VM? This post on nested VMs over at the VMware Communities site has some very useful information.
  • Paul Fazzone (who I believe is a product manager for the Nexus 1000V) points out a good point-counterpoint article with Bob Plankers and David Davis that discusses the benefits and drawbacks of the Cisco Nexus 1000V. Both writers make excellent points; I guess the real conclusion is that both options offer value for different audiences. Some organizations will prefer the VMware vSwitch (or Distributed vSwitch); others will find value in the Cisco Nexus 1000V. Choice is a beautiful thing.
  • Jason Boche published some performance numbers for the EMC Celerra NS-120 that he’s recently added to his home “lab” (I use the term “lab” rather loosely here, considering the amount of equipment found there). Not surprisingly, Fibre Channel won out over software iSCSI and NFS, but Jason’s numbers showed a larger gap than many expected. I may have to repeat these tests myself in the EMC lab in RTP to see what sorts of results I see. If only I still had the NS-960 that I used to have at ePlus….sigh.
  • Joep Piscaer has a good post on Raw Device Mappings (RDMs) that definitely worth a read. He’s pulled together a good summary of information on RDMs, such as requirements, limitations, use cases, and frequently asked questions. Good job Joep!
  • Ivo Beerens has a pretty detailed post on multipathing best practices for VMware vSphere 4 with HP EVA storage. The recommendation is to use Round Robin with ALUA and to reduce the IOPS limit to 1. Ivo also presents a possible workaround to the IOPS “random value” bug that Chad Sakac discussed in this post some time ago.
  • Here’s yet another great diagram by Hany Michael, this time on ESX memory management and monitoring.
  • This post tells you how to modify your VMware Fusion configuration files to assign IP addresses for NAT-configured VMs. If you’re familiar with editing dhcpd.conf on a Linux system, the information found here on customizing Fusion should look quite familiar.
  • Back in 2007, I wrote a piece on using link state tracking in blade deployments. This post wasn’t necessarily virtualization focused, but certainly quite applicable to virtualization environments. Recently I saw this article pop up on using link state tracking with VMware ESX environments. It’s good to see more people recommending this functionality, which I feel is quite useful.
  • Congratulations to Mike Laverick of RTFM, who this past week announced that TechTarget is acquiring RTFM and its author, much like TechTarget acquired BrianMadden.com (and its author) last year. Is this a new trend for technical blog authors—build up a readership and then “sell it off” to a digital media company?

Here are some additional links that I stumbled across, but for which I haven’t yet fully assimilated or processed. You might see some more in-depth blog posts about these in the near future as they work their way through my consciousness.

Lab Experiment: Hypervisors (Virtualization Review)
The Backup Blog: Avamar and VMware Backup Revisited
VMware vSphere Capacity IQ Overview – I’m Impressed!

Well, that wraps it up for now. Thanks for reading and feel free to speak out in the comments below.

Tags: , , , , ,

I’ve been doing a pretty fair amount of work recently with the Cisco Nexus 5000 series of switches, as evidenced by the flurry of Nexus-related articles:

Connecting Nexus 5000 to Older Gigabit Ethernet Switches
Setting Up FCoE on a Nexus 5000
FCoE and VLAN Trunking on Nexus 5000

One thing I hadn’t yet documented was how to enable jumbo frames on a Nexus 5000. Since jumbo frames are now officially supported for VMkernel traffic with VMware vSphere, the combination of jumbo frames and 10Gb Ethernet is an attractive one. I’ve covered the ESX/ESXi side (ordinary vSwitches here and distributed vSwitches here), but here’s the Nexus side.

The commands are pretty straightforward, and I’ve included the commands for both NX-OS 4.0 and NX-OS 4.1 (they are different between versions). Important note: if you enabled jumbo frames under NX-OS 4.0 and then upgraded the switch to version 4.1, you’ll need to re-do your jumbo frame configuration.

For NX-OS 4.1, the commands to enable jumbo frames are:

switch(config)# policy-map type network-qos jumbo
switch(config-pmap-nq)# class type network-qos class-default
switch(config-pmap-c-nq)# mtu 9216
switch(config-pmap-c-nq)# exit
switch(config-pmap-nq)# exit
switch(config)# system qos
switch(config-sys-qos)# service-policy type network-qos jumbo

Now, contrast the commands above with the following commands, which you would have used to enable jumbo frames on NX-OS 4.0:

switch(config)# policy-map jumbo
switch(config-pmap)# class class-default
switch(config-pmap-c)# mtu 9216
switch(config-pmap-c)# exit
switch(config)# system qos
switch(config-system)# service-policy jumbo

The end result of these differences is this: if you upgrade NX-OS from 4.0 to 4.1, then your jumbo frames configuration will go away, and you’ll need to enter the commands for version 4.1 in order to enable jumbo frame support again. This little gotcha caused me quite a headache when my NFS-based datastores suddenly went offline after the NX-OS upgrade.

More information on the necessary commands can be found here for version 4.0 and here for version 4.1.

Tags: , ,

VMware, Cisco, and EMC made their official announcement of the VCE Coalition and the joint venture Acadia this morning. You can read one of the press releases here via MarketWire.

Acadia is interesting, but it really isn’t the meat of the announcement, in my opinion. The real substance of the matter is the nature of the coalition. There are many interesting questions/thoughts circling in my head right at the moment:

  • What impact will this have on VMware’s relationship(s) with HP, IBM, and Dell? “Throwing their hat in the ring” with Cisco’s UCS, so to speak, may greatly endanger VMware’s much larger (with respect to revenue) relationships with other OEMs. What will happen to VMware if those OEMs “throw their hat in the ring” with Microsoft and Hyper-V? This is not a good place to be.
  • The acrimonious Cisco-HP relationship adds further fuel to the concerns over VMware’s close alliance with Cisco’s computing platform.
  • Does this new coalition signal a move away from the “arms-length” relationship between EMC and VMware, a move that some (competitors, notably) have been talking about for some time? If so, what danger does that put VMware in with regards to storage relationships?
  • It seems to me that VMware has the most to lose here. What does EMC lose if this doesn’t go well? Nothing, really. What about Cisco? Nothing, really. VMware, on the other hand…well, it could be ugly.
  • What does this coalition offer that the three companies couldn’t deliver without the coalition? Why risk important relationships? This is a big question in my mind. Lots of technology companies have delivered validated designs without any sort of formal coalition. Why is one necessary in this case?
  • On the other end of the spectrum—keeping Acadia out of the picture for the moment—is this “new coalition” really anything more than what the three companies have already been doing? Is this really anything more than each of the companies dedicating resources to this effort? I know from my own direct interaction with at least one of these vendors that resources had already been dedicated to the VCE technology intersection before any sort of formal announcement. So, does this formal announcement really mean anything at all?

I don’t have any answers (yet), but you can at least read my thoughts—and contribute back to them via the comments—without having to pay $499 to some analyst firm.

By the way, if you’d like some other viewpoints on this matter, here are a couple from opposing viewpoints:

NetApp – Jay’s Blog: The Importance of Being Open
Chuck’s Blog: Announcing the VCE Coalition

Feel free to speak up in the comments below (courteous comments only, please, and be sure to include full vendor disclosure where appropriate). Thanks!

Tags: , , , , , , , ,

In my earlier post on how to configure FCoE on a Nexus 5000, one of the readers suggested in the comments that it was necessary to have the interfaces in VLAN trunk mode via the switchport mode trunk command. I didn’t pay that much attention to it because the interfaces were indeed in VLAN trunk mode.

Fast forward to yesterday, when I was troubleshooting a problem between a Gen2 QLogic CNA and the Nexus 5010 in my lab (I tweeted about it). Although the Ethernet side of the CNA works just fine, the CNA refuses to bring up an FCoE connection. In the process of troubleshooting, Brad Hedlund (check his outstanding web site) suggested to me in a Twitter direct message that I should double-check the VLAN trunking status of the interface. That part I’d already heard from the reader who commented on the first post, but the next part was new to me (emphasis mine):

Gen2 requires ‘switchport mode trunk’ on the 5K. Gen1 doesn’t. Also make sure FCoE VLANs are allowed on the trunk.

Ah, now there’s something I hadn’t heard! That prompted me to do a bit of testing this morning (yes, I know I’m supposed to be studying for the VCDX Design Exam this afternoon). In my testing, I confirmed that a Gen1 CNA (I’m using Gen1 Emulex CNAs) does not require VLAN trunking to be enabled on the Ethernet interface.

There does appear to be a “gotcha” though: if the Ethernet interface is in access mode, it’s access VLAN must be the same as the FCoE VLAN; otherwise, the vfc interface will report down.

In summary:

  • If you are using a Gen2 CNA, you must put the Ethernet interface in VLAN trunk mode.
  • If you are using a Gen1 CNA, the Ethernet interface may be in either access mode or trunk mode.
  • If the interface is in trunk mode, be sure that you have allowed the FCoE VLAN via the switchport trunk allowed vlan command.
  • If the interface is in access mode, be sure that you have placed the interface in the FCoE VLAN via the switchport access vlan command.

If there are any other subtleties or nuances I’ve missed, please post them in the comments below so that future readers will benefit. Thank you!

Tags: , , , , ,

Fibre Channel over Ethernet (FCoE) is receiving a great deal of attention in the media these days. Fortunately, setting up FCoE on a Nexus 5000 series switch from Cisco isn’t too terribly complicated, so don’t be too concerned about deploying FCoE in your datacenter (assuming it makes sense for your organization). Configuring FCoE basically consists of three major steps:

  1. Enable FCoE on the switch.
  2. Map a VSAN for FCoE traffic onto a VLAN.
  3. Create virtual Fibre Channel interfaces to carry the FCoE traffic.

The first step is incredibly easy. To enable FCoE on the switch, just use this command:

switch(config)# feature fcoe

The next part of the FCoE configuration is mapping a VSAN to a VLAN. What VSAN should you use? Well, if you are connecting to an existing Fibre Channel fabric, perhaps on a Cisco MDS switch, you’ll need to make sure that the VSANs between the Nexus and the MDS are appropriately matched. Otherwise, traffic on one VSAN on the Nexus won’t be able to reach devices on another VSAN on the MDS. If there’s enough demand, I’ll post a quick piece on this step as well.

Note that this FCoE VSAN-to-VLAN mapping is a required step; if you don’t do this, the FCoE side of the interfaces won’t come up (as you’ll see later in this post). Assuming the VSAN is already defined, perform these steps to map the VSAN to a VLAN:

switch(config)# vlan XXX
switch(config-vlan)# fcoe vsan YYY
switch(config-vlan)# exit

Obviously, you’ll want to substitute XXX and YYY for the correct VLAN and VSAN numbers, respectively.

After you’ve enabled FCoE and mapped FCoE VSANs onto VLANs, then you are ready to create virtual Fibre Channel (vfc) interfaces. Each physical Nexus port that will carry FCoE traffic must have a corresponding vfc interface. Generally, you will want to create the vfc interface with the same number as the physical interface, although as far as I know you are not required to do so. It just makes management of the interfaces easier. The commands to create a vfc interface look like this:

switch(config)# interface vfc ZZ
switch(config-if)# bind interface ethernet 1/ZZ
switch(config-if)# no shutdown
switch(config-if)# exit

At this point the vfc interface is created, but it won’t work yet; you’ll need to place it into an VSAN that is mapped to an FCoE enabled VLAN. If you don’t, the show interface vfc <number> command will report this (emphasis mine):

vfc13 is down (VSAN not mapped to an FCoE enabled VLAN)

As I mentioned earlier, if you haven’t mapped the FCoE VSAN onto a VLAN, you won’t be able to fix this problem. If you have mapped the FCoE VSAN onto a VLAN, then you only need to assign the vfc interface to the appropriate VSAN with these commands:

switch(config)# vsan database
switch(config-vsan-db)# vsan <number> interface vfc <number>
switch(config-vsan-db)# exit

At this point, the vfc interface will report up, and you should be able to see the host’s connection information with the show flogi database command.

From this point—assuming that your storage is attached to a traditional Fibre Channel fabric, which is likely to be the case in the near future—you only need to create zones with the WWNs of the FCoE-attached hosts in order to grant them access to the storage. Refer to my posts on creating zones and managing zones on a Cisco MDS for more information on this task.

In my own experience, once FCoE was properly configured on the Nexus 5000 switch, then creating zones and zonesets on the Cisco MDS Fibre Channel switch and creating and masking LUNs on the Fibre Channel-attached storage is very straightforward. This, as has been stated on several previous occasions, is one of the strengths of FCoE: it’s compatibility with existing Fibre Channel installations is outstanding.

Feel free to submit any questions or clarifications in the comments below.

Tags: , , , , ,

« Older entries § Newer entries »