Networking

This category contains information on networking and networking-related technologies or vendors.

A flurry of virtualization-related product announcements flew into my Inbox today, thoroughly disrupting the empty Inbox I’d cultivated before the show. Anyway, I thought readers might be interested in some of the announcements, so here they are:

  • Akorri announced they’ve achieved VMware Ready status with their BalancePoint product. If you’re at VMworld and want to talk to Akorri, stop by booth 1331.
  • Similarly, Avere Systems has also been awarded VMware Ready status for its FXT 2700 appliance. Avere is also at VMworld in San Francisco, but I don’t have their booth number available to me.
  • Start-up company DeskStream has launched a product called Dynamic Virtual Desktop (yes, the acronym is DVD). It’s a “Desktop as a Service” product, according to their information. No word on whether DeskStream is at the VMworld conference. Follow this link for the full launch announcement.
  • Yet another company, CompuWare, has gotten VMware Ready status for CompuWare Vantage. As with DeskStream, I don’t have any indication as to whether CompuWare is at the VMworld conference.
  • I continue to be impressed by security startup HyTrust. Their latest announcement, HyTrust Cloud Control, brings strong authentication, role-based access control, and integration between HyTrust Appliance and VMware vCloud Director.
  • BLADE has announced VMready 3.0 with Virtual Vision, which allows physical networks to “see” virtual machines as they migrate (or are migrated) around the data center. At first glance, it kind of sounds like Arista’s VM Tracer, but I have a meeting with BLADE later this week and intend to find out more about the product. I’ll post more after that meeting.
  • EMC’s RSA division is also announcing the RSA Solution for Cloud Security and Compliance. This solution integrates technologies from Archer into a solution that is intended to help customers have greater confidence that their environments are properly secured and audited according to standards and policies. The full press release is also available here.

I think that’s about it for now. More VMworld 2010 coverage to come, so stay tuned!

Tags: , , , , , ,

Welcome to Technology Short Take #2, a collection of links, thoughts, ideas, and items pertaining to data center technologies—virtualization, networking, storage, and security. I hope you find something useful or interesting!

  • The release of FLARE 30 and DART 6 by EMC (formally announced last week) introduces some new concepts and new functionality. Matt Hensley recently did a write-up on some of the new functionality in this post on virtual provisioning, storage pools, and FLARE 30. It’s worth a read if you aren’t already familiar with these technologies and need a primer.
  • If you are looking for the definitive guide on connectivity between various VMware vSphere components and the TCP/UDP ports required, you need only look here. Great information!
  • Here’s a great guide from Cisco on deployment options when deploying 10 Gigabit Ethernet on VMware vSphere 4.0 with the Nexus 1000V or the VMware vNetwork Distributed Switch. I’ve read through it, but I’ve added it to my list of documents to go back and study more carefully; there’s lots of useful information in here.
  • Way back in March Dave Convery posted this article on limitations with VMware vShield Zones. While re-reading that article today, I noted in the comments that the Nexus 1000V has a feature called Virtual Service Domains that help address some of the limitations of vShield Zones (at that time). As pointed out in the comments, this makes vShield Zones usable in two NIC scenarios such as with Cisco UCS. If anyone has any additional links on Virtual Service Domains, please share them in the comments. This is a topic that I think needs some additional attention.
  • This article is a good breakdown of the differences in storage identifiers between ESX 3.x and ESX 4.1.
  • Jeff Woolsey at Microsoft finally wraps up his series of articles on Hyper-V Dynamic Memory with Part 6. I’ve been reading this series pretty faithfully as Jeff systematically lays out the various ways in which memory is handled in a virtualization scenario, and I’ve been consistently struck by the impression that Jeff was working really hard to distinguish what Microsoft was doing with Hyper-V from what VMware does with ESX/ESXi. In the end, though, I can’t help but see all the similarities between the two. Dynamic Memory allocates additional memory to a VM as it needs it (much the same way ESX/ESXi does by allocating memory only as requested by the VM) and reclaims free pages from the VMs (just like ESX/ESXi reclaims idle pages via idle page reclamation). When under memory pressure, Hyper-V might force the guests to page out to disk; ESX/ESXi’s memory balloon driver achieves the same effect. What’s missing, obviously, is that with Hyper-V the hypervisor itself won’t swap pages out to disk (ESX/ESXi will do this under extreme circumstances). Am I missing something, or is Microsoft’s Dynamic Memory a lot more like VMware’s memory management technologies than Microsoft wants to admit? Feel free to enlighten me (courteously and with full disclosure) in the comments if I’m missing something.
  • Via Geert Verbist’s site, I found this article on application consistent quiescing via VMware’s VSS integration in VMware Tools. (For more information on VSS support within VMware Tools, check out my liveblog from Partner Exchange earlier this year.) This is good to hear, but what’s still not clear is whether the application consistent snapshots will truncate transaction logs. If anyone has more information, speak up in the comments.
  • I think I pointed this out a week or two ago on Twitter, but I thought I’d mention here at well. If you ever need to help decode which WWPNs map to which ports on an EMC CLARiiON array, this article is quite helpful. Anyone have matching articles for EMC Symmetrix, NetApp, HP, HDS, or other arrays?
  • With the formal announcement by VMware that vSphere 4.1 will be the last major release that includes ESX, ESXi is naturally getting much more attention. With that, there’s been a flurry of ESXi-related articles:
    Using vMA As Your ESXi Syslog Server
    The Migration From ESX to ESXi is Happening: Moving Configurations, Part 1
    The Migration from ESX to ESXi is Happening: Moving Configurations, Part II
    My VMware ESXi Installation Checklist
    Virtually Ghetto: ESXi 4.1 - Major Security Issue (also documented here in the VMware KB)
    ESXi 4.1 - Major Security Issue - The Sequel and the Workaround
    ESXi 4.1 Active Directory Integration
  • If you’re into Cisco UCS but like Hyper-V instead of VMware vSphere, Cisco has a white paper on Cisco UCS with Hyper-V for delivery of virtualized Exchange 2010.
  • I’m a command-line junkie, so I liked this article on how to put an ESX host into maintenance mode from the CLI.
  • For those seeking to get up to speed on the Nexus 7000 switches, “Fryguy” posted some training documents on his site. I haven’t read them (yet), but they’re on my list of documents to read (a list that grows ever longer…)

I guess that will do it for this time around. I hope that you’ve found something useful and, as always, feel free to add more useful links or tidbits in the comments. Thanks for reading!

Tags: , , , , , , , , ,

I posted a tweet earlier today that asked this question:

If I shouldn’t use “ip default-network” because it’s classful, then should I redistribute a static default route?

What prompted this question was some work I was doing earlier today in preparation for my CCNA exam. I had a five-site hub-and-spoke network in GNS3 running EIGRP, with lightweight OpenBSD VMs attached behind each router so that I could test end-to-end connectivity (i.e., ping a host behind one router from a host behind another router). This configuration is working fine.

Then I decided I’d take this setup and hide it behind a Vyatta VM performing NAT and see if I could connect it to the rest of my home network. The Vyatta stuff works fine, but now I’m faced with the prospect of configuring this self-contained little environment with a default route that points to the Vyatta. The Vyatta, in turn, points to the physical firewall protecting the home network from the nasty Internet. This configuration doesn’t seem at all too far-fetched from a realistic deployment where an enterprise network would need a default route out to the Internet, presumably through a firewall performing network address translation.

So what’s the best way to do it? I’ve read a couple of articles (older ones, since that’s all that seems to be available) saying that the ip default-network shouldn’t be used because it’s classful. To be honest, I’m not sure I fully understand the behavior of that command anyway, but if I’m not supposed to use that then do I just set a static route and redistribute that into EIGRP for distribution to the rest of the routers?

Sorry, I’m still learning here…

Tags: ,

This is one article in a series of articles focused toward new users. Some other New User’s Guide articles include:

This particular article is a follow-up of sorts to the first article listed above. While that article focused on virtual networking with VMware ESX, this article focuses on virtual networking with VMware ESXi. Given that VMware’s stated focus is on VMware ESXi moving forward, I thought this article would be helpful and timely.

For new users who are seeking a thorough explanation of how VMware ESX/ESXi networking functions, I’ll recommend a series of articles by Ken Cline titled The Great vSwitch Debate. Ken goes into a great level of detail. Go read that, then you can come back here.

All of the commands presented in this article were testing using VMware vSphere 4.1. The environment consisted of hosts running VMware ESXi 4.1 being managed by VMware vCenter Server 4.1. For CLI access, I used the vSphere Management Assistant (vMA) virtual appliance, deployed via OVF.

The majority of all the networking configuration you will need to perform on VMware ESXi boils down to just a few commands:

  • vicfg-vswitch: You will use this command to manipulate virtual switches (vSwitches) and port groups.
  • vicfg-vmknic: You will use this command to create, modify, or delete VMkernel NICs on the VMware ESXi hosts.
  • vicfg-nics: You will use this command to view (and potentially manipulate) the physical network interface cards (NICs) in a VMware ESXi host.

The tasks that you’ll actually perform using this commands are pretty straightforward:

  1. Creating, configuring, and deleting vSwitches
  2. Creating, configuring, and deleting port groups
  3. Creating, configuring, and deleting VMkernel NICs

I’ll start with a few prerequisites that are necessary due to the fact that you are using a remote CLI to access the VMware ESXi hosts.

As you can see from the list above, all the commands you’re going to use are the vicfg-* commands. All of these commands have some standard parameters they require in addition to the task-specific parameters. To make things a bit simpler for you, I’ll recommend that you set persistent values (persistent for the current vMA session, at least) to simplify the commands later. Here are the values I recommend you establish:

  • First, set the value of the VI_SERVER variable to be the fully qualified domain name of the vCenter Server computer. Use the bash export command to set this variable, like this:
     
    export VI_SERVER=vcenter-server.domain.com
     
    Setting this variable now means that none of the vicfg-* commands will need to have this parameter specified. Since it’s likely that you’ll consistently work with one specific instance of vCenter Server, then this is a pretty safe variable to set.
  • In the absence of using Active Directory integration (which is a far cleaner choice, but one which we’ll reserve for a future article), set the VI_USERNAME variable to the name of the user account you’ll use to authenticate against vCenter Server. Again, use the export command as outlined in the previous bullet.

Now that you have some basics established, I’ll move on to creating, configuring, and deleting vSwitches.

Creating, Configuring, and Deleting vSwitches

You’ll use the vicfg-vswitch command for the majority of these tasks. Unless I specifically indicate otherwise, all the commands, parameters, and arguments are case-sensitive. For all these vicfg-* commands, you will get prompted for the password to the user account you defined when you set the value of the VI_USERNAME variable.

To create a vSwitch, use this command:

vicfg-vswitch -h <ESXi hostname> -a <vSwitch Name>

To link a physical NIC to a vSwitch—which is necessary in order for the vSwitch to pass traffic onto the physical network or to receive traffic from the physical network—use this command:

vicfg-vswitch -h <ESXi hostname> -L <Physical NIC> <vSwitch Name>

In the event you don’t have information on the physical NICs, you can use this command to list the physical NICs:

vicfg-nics -h <ESXi hostname> -l (lowercase L)

Conversely, if you need to unlink (remove) a physical NIC from a vSwitch, use this command:

vicfg-vswitch -h <ESXi hostname> -U <Physical NIC> <vSwitch Name>

To change the Maximum Transmission Unit (MTU) size on a vSwitch, use this command:

vicfg-vswitch -h <ESXi hostname> -m <MTU size> <vSwitch Name>

To delete a vSwitch, use this command:

vicfg-vswitch -h <ESXi hostname> -d <vSwitch Name>

Creating, Configuring, and Deleting Port Groups

As with virtual switches, the vicfg-vswitch is the command you will use to work with port groups. Once again, unless I specifically indicate otherwise, all the commands, parameters, and arguments are case-sensitive.

To create a port group, use this command:

vicfg-vswitch -h <ESXi hostname> -A <Port Group Name> <vSwitch Name>

To set the VLAN ID for a port group, use this command:

vicfg-vswitch -h <ESXi hostname> -v <VLAN ID> -p <Port Group Name> <vSwitch Name>

To delete a port group, use this command:

vicfg-vswitch -h <ESXi hostname> -D <Port Group Name> <vSwitch Name>

To view the current list of vSwitches, port groups, and uplinks, use this command:

vicfg-vswitch -h <ESXi hostname> -l (lowercase L)

Creating, Configuring, and Deleting VMkernel NICs

To work with ESXi’s VMkernel NICs, you’ll primarily use the vicfg-vmknic command. As in the previous sections, all commands are case-sensitive unless I specifically indicate otherwise, and all commands assume you’ve defined the VI_SERVER and VI_USERNAME variables.

To create a new VMkernel NIC, use this command:

vicfg-vmknic -h <ESXi hostname> -a -i <VMkernel NIC IP address> -n <Subnet mask> <Port group>

To delete a VMkernel NIC, use this command:

vicfg-vmknic -h <ESXi hostname> -d <Port group>

To enable vMotion on an already-created VMkernel NIC:

vicfg-vmknic -h <ESXi hostname> -E <Port group>

There are more networking-related tasks that you can perform from the CLI, but for a new user these commands should handle the lion’s share of all the networking configuration. Good luck with your ESXi environment!

Tags: , , , ,

The topic of vMotion, it’s practicality, and Layer 2 adjacency for vMotion has been a topic I’ve visited a few times over the last several months. The trend got kicked off with a post on vMotion reality, in which I attempted to debunk an article claiming vMotion was only a myth. The series continued with a discussion of the practicality of vMotion, where I again discussed the various Layer 2 requirements for vMotion.

In the vMotion practicality article, reader Paul Pindell, an employee of F5 Networks, discusses the networking requirements for vMotion. To quote from his comment:

Notice that there is no requirement for the vMotion VMkernel interfaces of the ESX(i) hosts to have what was termed Layer 2 adjacency. The vMotion VMkernel interfaces are not required to be on the same subnet, VLAN, nor on the same L2 broadcast domain. The vMotion traffic on VMkernel interfaces is routable. IP-based storage traffic on VMkernel interfaces is also routable. Thus there are no L2 adjacency requirements for vMotion to succeed.

I was intrigued by this statement, so I contacted Duncan Epping (of Yellow Bricks fame) and discussed the matter with him. Duncan has also posted on this topic on his site as well; both his post and my post are the result of our discussion and collaboration around this matter.

So is Layer 2 adjacency for vMotion a requirement, or not? In the end, the answer is that Layer 2 adjacency for VMkernel interfaces configured for vMotion is not required; vMotion over a Layer 3 interface will work. The caveat is that routed vMotion, as it has sometimes been called, hasn’t been through the extensive VMware QA process and therefore is not yet supported. (Please don’t mistake my use of the word “yet” as any sort of commitment by VMware to actually support it.)

In summary, then: vMotion across two different subnets will, in fact, work, but it’s not yet supported by VMware. As additional information becomes available—as Duncan indicated, the VMware KB article is going to be updated to avoid misunderstanding—I’ll update this post accordingly.

Tags: , , , ,

I’ve decided to discontinue the Virtualization Short Takes series and replace it with a new series that incorporates items from the networking and storage realms as well. In truth, I was already doing a little bit of that anyway, but this will expand and formalize the coverage a bit. I’d love to hear your feedback on this new series, to help me ensure that I’m providing useful and pertinent information and to help me ensure that I’m presenting it in the most efficient way. So, please speak up in the comments if you have any suggestions for improvement. Thanks!

And with that brief introduction, welcome to Technology Short Take #1!

  • You can tell just how far behind I’ve gotten with this sort of post when one of the items I’m mentioning was published almost three months ago. Still, one phrase in this article by Frank Denneman on VM memory overhead really caught my eye:

    Please be aware of the fact that memory overheads are growing with each new release of ESX, so keep this in mind when upgrading to a new version.

    What does this mean? Why are VM memory overheads increasing, and is this something to be concerned over? I can certainly understand the need to add features, but is the drive to add features to the hypervisor also driving bloat and making it less efficient in some aspects?

  • Duncan Epping wrote a great post about a month ago on new behavior in vSphere 4.1 with respect to HA/DRS and flattened shares. This behavior is yet another reason you want to make sure that you really understand what you’re doing when you assign reservations, limits, and shares to your VMs and resource pools. Both Duncan and his cohort-in-crime Frank, along with a few others, have been publishing lots of resource allocation-related posts:
    Which host is selected for an HA initiated restart?
    Memory reclamation, when and how?
    Reservations and CPU scheduling
    The Math Behind the DRS Stars
  • Need to flip the paths on your datastores over to a different HBA so you can perform some SAN maintenance? Leo Raikhman has a couple of scripts that can help with performing SAN fabric maintenance on ESX.
  • Larry Touchette at NetApp has a good write-up on DRS host affinity rules in vSphere 4.1. I have to take exception to Larry’s tying of DRS host affinity rules to NetApp MetroCluster (DRS host affinity rules have so many more applications than just stretched clusters!), but that’s a minor nit. For more information on DRS host affinity rules, you can also check out this post.
  • This VMware KB article looks like it could be pretty handy if you are using the Nexus 1000V and need to migrate to a new vCenter Server instance.
  • Dave Alexander does some “thinking out loud” with a post on private VSANs. In the post he asks why we don’t use PVSANs in the FC world in the same way we use PVLANs in the networking world. It’s an interesting thought, but it sounds like all it accomplishes is simplifying the creation of zones. Surely there are easier ways to simplify the creation of zones than creating an administrative construct like PVSANs.
  • More “thinking out loud” comes from Didier Pironet in his post on enhancements to VMware DRS/HA/FT. In his post, Didier suggests the incorporation of hardware information into DRS/HA/FT to allow for the creation of policy driven decisions that help decide where to place VMs based on hardware attributes. This is a useful line of thought and one that I think vendors will need to explore in order to allow service providers to offer various differentiated levels of service.
  • I found this brief post quite helpful, as I was quite confused as to how the programming language had anything to do with routing. Turns out I was using the wrong acronym, and LISP instead stands for Location and Identity Separation Protocol. Ah, that makes a lot more sense now!
  • Aside from a brief mention here, I hadn’t seen any references to the fact that VMware released VMware Studio 2.1 along with vSphere 4.1.
  • Fellow EMC vSpecialist Simon Seagrave recently published a good article on when/how UUIDs are generated for virtual machines.

Because I’m so far behind, I have a ton of other links that I’ve been collecting. They are all good articles and worth reading if you can spare the time. Here are a few that stand out:

How does “das.maxvmrestartcount” work?
Yes, You Really *Can* Run Tier-1 Enterprise Applications on vSphere
How to Configure Likewise “Open” Integration on vMA
Got Network I/O Control?
VMware HA Isolation Response Bug
Cisco, VMware: Nexus 1000V Tracking VM Interface Errors

I guess that’s going to do it for this time around. As I said earlier, I’d love to hear your feedback, positive or negative, as well as any suggestions for improvement. Thanks for reading!

Tags: , ,

About a month ago I posted an article titled The vMotion Reality. In that article, I challenged the assertion that vMotion was a myth, not widely implemented in data centers today due to networking complexity and networking limitations.

In a recent comment to that article, a reader again mentions networking limitations—in this instance, the realistic limits of a large Layer 2 broadcast domain—as a gating factor for vMotion and limiting its “practical use” in today’s data centers. Based on the original article’s assertions and the arguments found in this comment, I’m beginning to believe that a lot of people have a basic misunderstanding of the Layer 2 adjacency requirements for vMotion and what that really means. In this post, I’m going to discuss vMotion’s Layer 2 requirements and how they interact (and don’t interact) with other VMware networking functionality. My hope is that this article will provide a greater understanding of this topic.

First, I’d like to use a diagram to help explain what we’re discussing here. In the diagram below, there is a single ESX/ESXi host with a management interface, a vMotion interface, and a couple of network interfaces dedicated to virtual machine (VM) traffic.

VLAN Behaviors with VMware ESX/ESXi

As you can see from this highly-simplified diagram, there are three basic types of network interfaces that ESX/ESXi uses: management interfaces, VMkernel interfaces, and VM networking interfaces. Each of them is configured separately and, in many configurations, quite differently.

For example, the management interface (in VMware ESX this is the Service Console interface, in VMware ESXi this is a separate instance of a VMkernel interface) is typically configured to connect to an access port with access to a single VLAN. In Cisco switch configurations, the interface configuration would look something like this:

interface GigabitEthernet 1/1
  switchport mode access
  switchport access vlan 27
  spanning-tree portfast

There might be other commands present, but these are the basic recommended settings. This makes the management interfaces on an ESX/ESXi host act, look, feel, and behave just exactly like the network interfaces on pretty much every other system in the data center. Although VMware HA/DRS cluster design considerations might lead you toward certain Layer 2 boundaries, there’s nothing stopping you from putting management interfaces from different VMware ESX/ESXi hosts in different Layer 3 VLANs and still conducting vMotion operations between them.

So, if it’s not the management interfaces that are gating the practicality of vMotion in today’s data centers, it must be the VM networking interfaces, right? Not exactly. Although the voices speaking up against vMotion in this online discussion often cite Layer 3 VLAN concerns as the primary problem with vMotion—stating, rightfully so, that the IP address of a migrated virtual machine cannot and should not change—these individuals are overlooking the recommended configuration for VM networking interfaces in a VMware ESX/ESXi environment.

Physical network interfaces in a VMware ESX/ESXi host that will be used for VM networking traffic are most commonly configured to act as 802.1Q VLAN trunks. For example, the Cisco switch configuration for a port connected to a network interface being used for VM networking traffic would look something like this:

interface GigabitEthernet1/10
  switchport trunk encapsulation dot1q
  switchport mode trunk
  switchport trunk allowed vlan 1-499
  switchport trunk native vlan 499
  spanning-tree portfast trunk

As before, some commands might be missing or some additional commands might be present, but this gives you the basic configuration. In this configuration, the VM networking interfaces are actually capable of supporting multiple VLANs simultaneously. (More information on the configuration of VLANs with VMware ESX/ESXi can be found in this aging but still very applicable article.)

In practical use what this means is that any given VMware ESX/ESXi host could have VMs running on it that exist on a completely different and separate VLAN than the management interface. In fact, any given VMware ESX/ESXi host might have multiple VMs running across multiple separate and distinct VLANs. And as long as the ESX/ESXi hosts are properly configured to support the same VLANs, you can easily vMotion VMs between ESX/ESXi hosts when those VMs are in different VLANs. So, the net effect is that ESX/ESXi can easily support multiple VLANs at the same time for VM networking traffic, and this means that vMotion’s practical use isn’t gated by some inherent limitation of the VM networking interfaces themselves.

Where in the world, then, does this Layer 2 adjacency thing keep coming from? If it’s not management interfaces (it isn’t), and it’s not VM networking interfaces (it isn’t), then what is it? It’s the VMkernel interface that is configured to support vMotion. In order to vMotion a VM from one ESX/ESXi host to another, each host’s vMotion-enabled VMkernel interface has to be in the same IP subnet (i.e., in the same Layer 2 VLAN or broadcast domain). Going back to Cisco switch configurations, a vMotion-enabled VMkernel port will be configured very much like a management interface:

interface GigabitEthernet 1/5
  switchport mode access
  switchport access vlan 37
  spanning-tree portfast

This means that the vMotion-enabled VMkernel port is just an access port (no 802.1Q trunking) in a single VLAN.

Is this really a limitation on the practical use of vMotion? Hardly. In my initial rebuttal of the claims against vMotion, I pointed out that because it is only the VMkernel interface that must share Layer 2 adjacency, all this really means is that a vMotion domain is limited to the number of vMotion-enabled VMkernel interfaces you can put into a single Layer 2 VLAN/broadcast domain. VMs are unaffected, as you saw earlier, as long as the VM networking interfaces are correctly configured to support the appropriate VLAN tags, and the management interfaces do not, generally speaking, have any significant impact.

Factoring in the consolidation ratio, as I did in my earlier post, and you’ll see that it’s possible to support very large numbers of VMs spread across as many different VLANs as you want with a small number of ESX/ESXi hosts. Consider 200 ESX/ESXi hosts—whose vMotion-enabled VMkernel interfaces would have to share a broadcast domain—with a consolidation ratio of 12:1. That’s 2,400 VMs that can be supported across as many different VLANs simultaneously as we care to configure. Do you want 400 of those VMs in one VLAN while 300 are in a different VLAN? No problem, you only need to configure the physical switches and the virtual switches appropriately.

Let’s summarize the key points here:

  1. Only the vMotion-enabled VMkernel interface needs to have Layer 2 adjacency within a data center.
  2. Neither management interfaces nor VM networking interfaces require Layer 2 adjacency.
  3. VM networking interfaces are easily capable of supporting multiple VLANs at the same time.
  4. When the VM networking interfaces on multiple ESX/ESXi hosts are identically configured, VMs on different VLANs can be migrated with no network disruption even though the ESX/ESXi hosts themselves might all be in the same VLAN.
  5. Consolidation ratios multiply the number of VMs that can be supported with Layer 2-adjacent vMotion interfaces.

Based on this information, I hope it’s clearer that vMotion is, in fact, quite practical for today’s data centers. Yes, there are design considerations that come into play, especially when it comes to long-distance vMotion (which requires stretched VLANs between multiple sites). However, this is still an emerging use case; the far broader use case is within a single data center. Within a single data center, all the key points I provided to you apply, and there is no practical restriction to leveraging vMotion.

If you still disagree (or even if you agree!), feel free to speak up in the comments. Courteous and professional comments (with full disclosure, where applicable) are always welcome.

Tags: , , , , ,

Some time ago I posted a “how to” article on how to configure FCoE on a Nexus 5000 switch. At that time, I did not put the Nexus 5000 into NPV mode but rather connected it to a Cisco MDS Fibre Channel switch without using NPV. In this entry, I’d like to follow up on that article and show you how to configure NPV on a Nexus 5000.

If you aren’t familiar with NPV (N_Port Virtualization) and how it’s different than NPIV (N_Port ID Virtualization), check out this article titled “Understanding NPIV and NPV”; It should help clear things up. Because NPV makes the Nexus 5000 look like an NPIV-enabled host, one potential use for NPV, as in this case, is when connecting the Nexus 5000 to a non-Cisco Fibre Channel switch. Using NPV helps ease interoperability concerns. In this instance, I’m connecting the Nexus 5000 to a Brocade Fibre Channel switch (actually an EMC Connectrix).

Note that I tested these instructions on a Nexus 5010 using both NX-OS 4.1(3)N2(1) as well as NX-OS 4.2(1)N1(1).

The first step is to enable NPV on the Nexus 5000. As far as I can tell, in order to enable NPV you must first enable FCoE using the feature command:

switch(config)# feature fcoe

This loads various Fibre Channel modules and makes possible other features, including NPV and NPIV. Enabling NPV erases the switch configuration and reboots the switch, so be sure you are connected via a console connection before enabling NPV with the feature command:

switch(config)# feature npv

Immediately after enabling NPV, the Nexus 5000 will reboot (you’re warned and given the option to proceed or cancel). The warning indicates that the switch’s configuration will be removed, but the minimally-configured switches I used in my testing retained their configuration. Granted, I hadn’t performed any Fibre Channel or FCoE configurations yet, so perhaps that’s the configuration to which the warning was referring.

Once NPV is enabled on the switch, you can then configure Fibre Channel uplink ports as NP ports (proxy N_ports); these are also referred to as external interfaces. To configure a Fibre Channel port as an NP port, use these commands:

switch# config t
switch(config)# interface fc slot/port
switch(config-if)# switchport mode np
switch(config-if)# no shut
switch(config-if)# exit
switch(config)# exit

You should then be ready to physically connect to the upstream Fibre Channel switch, which—if you recall correctly from my earlier NPV/NPIV post—needs to be NPIV-enabled. In this particular case, I was uplinking the Nexus 5010 to an EMC-rebranded Brocade switch (a Connectrix DS-300B running Fabric OS 6.1.0). To show whether the port on the Connectrix was enabled for NPIV, I used the portcfgshow command:

rtp-fc-sw-01:admin> portcfgshow port number

Look for the line that says “NPIV Capability”; the value should be reported as “ON”. If the value is not “ON”, you’ll need to use the portcfgnpivport command to enable NPIV on the specified port, like this:

rtp-fc-sw-01:admin> portcfgnpivport port number 1

The “1″ at the end of that command enables NPIV; a “0″ would disable NPIV for that port.

Once NPIV is enabled on the upstream Fibre Channel switch, when you physically connect the configured external interface then the Fibre Channel link should come up. I used the show int fc slot/number command on the Nexus to verify that the port was up; on the Connectrix, I used the portshow port command. In addition, I was also able to see the Nexus switch logged into the Fibre Channel fabric on the Connectrix using the nsshow command.

Once you have Fibre Channel connectivity via the external interfaces, then configuring FCoE to hosts connected to the Nexus follows the same set of instructions laid out in my earlier FCoE how-to article.

From that point on, it’s only a matter of configuring zones (see here for help with zones on a Cisco MDS) and presenting storage. Those are different posts for a different day…

Tags: , , , , , ,

This article is part 1 of a series of posts that provide some additional information on creating UCS service profiles. In this part of the series, I provide some guidelines on networking-related elements that should be created before you create a UCS service profile. By creating these elements before creating the service profiles or service profile template, the process of creating the service profile or service profile template is a bit smoother.

There are three key networking elements that I recommend defining before you create a UCS service profile:

  1. At least one MAC address pool
  2. All the VLANs that need to be visible/accessible within UCS
  3. At least one vNIC template

The steps for creating each of these elements, along with their location in UCSM, is found below.

Creating a MAC Address Pool

Location in UCSM: LAN tab - LAN > Pools > MAC Pools
Prerequisites: None

Here are the steps for creating a MAC address pool:

  1. Right-click on MAC Pools and select Create MAC Pool.
  2. Specify a name and description for the new MAC pool, then click Next.
  3. Click Add (with the green plus symbol) to add a new MAC block.
  4. Specify the first MAC address and the size of the MAC block. Keep in mind that MAC pools (and the MAC address blocks within them) are not shared across UCSM instances, so be sure to provide a way of distinguishing MAC addresses across multiple UCSM instances. For example, use the fourth hextet to denote a UCSM instance (01 for the first UCSM instance, 02 for the second, etc.). This will help generate unique MAC addresses across multiple UCSM instances.
  5. Click OK to add the MAC address block to the MAC pool.
  6. Repeat steps 3 through 5, as needed, to add more MAC address blocks to this MAC pool. When the MAC address pool is complete, click Finish to complete the creation of the new MAC pool.

Defining VLANs

Location in UCSM: LAN tab - LAN > LAN Cloud > VLANs
Prerequisites: None

  1. Right-click on VLANs and select Create VLAN(s).
  2. Specify a descriptive name for the VLAN.
  3. Specify whether the VLAN should be global to both UCS fabrics, specific to only one fabric or the other, or if the fabrics should be configured differently for the same VLAN. Generally, VLANs will be configured as Common/Global.
  4. Specify the VLAN ID(s) that are associated with this VLAN object in UCSM. Generally, each VLAN object in UCSM should only have a single VLAN ID associated with it. If you enter multiple VLAN IDs (like “70-75″ or “21-24,27″), UCSM will create separate VLAN objects for each VLAN ID specified. This makes the creation of multiple VLANs a single step.
  5. When the dialog box is complete, click OK to create the VLAN object.
  6. Repeat this process to create VLAN objects for all VLANs that should be available within this UCSM instance. Use multiple VLAN IDs, as outlined earlier, to streamline the creation of multiple VLANs at the same time.

Creating a vNIC Template

Location in UCSM: LAN tab - LAN > Policies > vNIC Templates
Prerequisites: MAC address pool, VLANs

  1. Right-click on vNIC Templates and select Create vNIC Template.
  2. Specify a name and a description for this vNIC template.
  3. Select the UCS fabric to which this vNIC template should be associated. Do not select Enable Failover when using VMware ESX/ESXi. Note that you will need to create vNIC templates for both fabrics in dual fabric configurations.
  4. Leave all targets selected.
  5. Select Updating to make this an updating template. As additional VLANs are added in the future, modifying this template will then automatically modify service profiles created from this template to make the new VLANs available to associated blades.
  6. Place a check mark next to each VLAN that should be visible to vNICs created from this vNIC template.
  7. Mark one of the selected VLANs as the native VLAN; traffic from this VLAN will be untagged traffic. (In VMware ESX/ESXi, this traffic would require a port group with a blank VLAN ID configured.)
  8. Select the MAC address pool created earlier.
  9. If necessary, choose values for QoS policy, network control policy, pin group, and stats threshold policy. In general, these values do not need to be set.
  10. When the dialog box is complete, click OK to create the vNIC template.
  11. If this is a dual fabric installation, repeat this process for the second UCS fabric.

In part 2 of the series, I’ll provide information on the storage-related elements that I recommend creating before you create a UCS service profile.

If you have any questions or clarifications, please feel free to post them in the comments.

Tags: , , , ,

Yesterday I completed the configuration of inter-VLAN routing (aka “router on a stick”, or RoaS) as part of my ongoing CCNA preparation. A couple people mentioned that they would find the configuration useful, so I’m posting what I have. This is by no means a comprehensive treatise on the subject; for that, you should look elsewhere. Google can find you lots of sites with more in-depth and detailed information on the reasons behind the necessary configuration.

There are two primary components in a RoaS configuration:

  1. The configuration of the VLANs and VLAN trunking port on the switch
  2. The configuration of subinterfaces on the router

I describe how to configure each component below.

VLANs and VLAN Trunking

How to create VLANs varies between various switch types. On some switches, you’ll use the vlan database command in privileged EXEC mode. On other switches, you will use the vlan <VLAN ID> command while in global configuration mode. Regardless of which method is necessary for your particular Cisco switch, you will want to ensure that the switch has all the necessary VLANs defined.

After the VLANs have been defined, then you will need to configure the switch port connected to the router as a VLAN trunk port. This is pretty well covered elsewhere, but here is a quick review of the commands (these commands assume port 15 on module 0, a Fast Ethernet port):

switch(config)# int fa0/15
switch(config-if)# switchport trunk encapsulation dot1q
switch(config-if)# switchport mode trunk
switch(config-if)# switchport trunk allowed vlan 1,71-75
switch(config-if)# exit
switch(config)# exit
switch#

A couple notes about these commands:

  • Some switches only accept 802.1Q VLAN encapsulation; therefore, the switchport trunk encapsulation dot1q command isn’t supported because that’s the only encapsulation supported. So, the support for this command will vary from switch to switch.
  • You will want to specify the correct VLANs for your environment in the switchport trunk allowed vlan command.

At this point, the switch is configured correctly; now it’s time to move to the router.

Subinterfaces on the Router

For each VLAN that needs to be routed, you will need to create a subinterface on the router. Creating a subinterface is pretty easy, the commands look something like this:

router(config)# int fa0/0.1
router(config-if)# encapsulation dot1q 1 native
router(config-if)# ip address 192.168.1.1 255.255.255.0
router(config-if)# exit
router(config)# exit
router#

As before, there are a few notes to consider about these commands:

  • The number of the subinterface (the “1″ in fa0/0.1 above) is only locally significant and doesn’t need to match the VLAN ID, but matching the VLAN ID makes it easier to associate the subinterface with its configured VLAN ID. Again, as stated earlier, you’ll need a separate subinterface for each VLAN that you want to route.
  • Only specify the native keyword on the encapsulation dot1q command if this is the native VLAN on the switch side as well. Otherwise, the trunk won’t form as expected.
  • The IP address specified here will be the IP address of the default gateway for that VLAN/subnet.

For the physical interface itself, the interface needs to be up (so don’t issue a shutdown command), but the interface does not need to have any IP address associated with it.

With this configuration in place, you should be able to route between the VLANs; just specify the IP address of the subinterface on the router for that VLAN as the default gateway of the systems on that VLAN and you should be good to go.

If I’ve missed anything glaring please speak up in the comments and let me know.

Tags: , , ,

« Older entries