Gestalt

Articles in this category are republished to the GestaltIT site.

I’m about halfway through the first day of Unified Computing System (UCS) training in San Jose, CA, and I’ve learned of what I think is a fairly significant limitation. The issue centers around what Cisco refers to as “northbound” traffic and how Fibre Channel over Ethernet (FCoE) is handled with northbound traffic.

Recall that a central part of UCS is the UCS 6100 series fabric interconnect. The 6100 series fabric interconnect has connectivity in two directions:

  • Southbound connectivity is connectivity aimed back at the fabric extenders in the blade chassis themselves.
  • Northbound connectivity is connectivity headed outside the UCS to other systems and networks.

All southbound traffic is 10Gbps Ethernet with FCoE. Northbound traffic can be 10Gbps Ethernet or Fibre Channel, but not FCoE. Based on the information I’ve been given (and if I’m incorrect please let me know in the comments), you cannot directly connect an FCoE-enabled storage array to a UCS. Even if your storage array has native FCoE interfaces, you can’t plug them into the UCS 6100 series fabric interconnects because that’s considered northbound traffic and you can’t use FCoE with northbound traffic.

I have a feeling customers who have purchased storage arrays with FCoE interfaces with the intention of hooking the arrays up directly to a UCS are going to be a bit upset when this information becomes more widely known.

If I’m working from incorrect or incomplete information, please feel free to speak up in the comments.

Tags: , , , , ,

A lot of the content on this site is oriented toward VMware ESX/ESXi users who have a pretty fair amount of experience. As I was working with some customers today, though, I realized that there really isn’t much content on this site for new users. That’s about to change. As the first in a series of posts, here’s some new user information on creating vSwitches and port groups in VMware ESX using the command-line interface (CLI).

For new users who are seeking a thorough explanation of how VMware ESX networking functions, I’ll recommend a series of articles by Ken Cline titled The Great vSwitch Debate. Ken goes into a great level of detail. Go read that, then you can come back here.

Before I get started it’s important to understand that, for the most part, the information in this article applies only to VMware ESX. VMware ESXi doesn’t have a Linux-based Service Console like VMware ESX, and therefore doesn’t have a readily-accessible CLI from which to run these sorts of commands. There is a remote CLI available, which I’ll discuss in a future post, but for now I’ll focus only on VMware ESX.

The majority of all the networking configuration you will need to perform on VMware ESX boils down to just a couple commands:

  • esxcfg-vswitch: You will use this command to manipulate virtual switches (vSwitches) and port groups.
  • esxcfg-nics: You will use this command to view (and potentially manipulate) the physical network interface cards (NICs) in the VMware ESX host.

Configuring VMware ESX networking boils down to a couple basic tasks:

  1. Creating, configuring, and deleting vSwitches
  2. Creating, configuring, and deleting port groups

I’ll start with creating, configuring, and deleting vSwitches.

Creating, Configuring, and Deleting vSwitches

You’ll primarily use the esxcfg-vswitch command for the majority of these tasks. Unless I specifically indicate otherwise, all the commands, parameters, and arguments are case-sensitive.

To create a vSwitch, use this command:

esxcfg-vswitch -a <vSwitch Name>

To link a physical NIC to a vSwitch—which is necessary in order for the vSwitch to pass traffic onto the physical network or to receive traffic from the physical network—use this command:

esxcfg-vswitch -L <Physical NIC> <vSwitch Name>

In the event you don’t have information on the physical NICs, you can use this command to list the physical NICs:

esxcfg-nics -l (lowercase L)

Conversely, if you need to unlink (remove) a physical NIC from a vSwitch, use this command:

esxcfg-vswitch -U <Physical NIC> <vSwitch Name>

To change the Maximum Transmission Unit (MTU) size on a vSwitch, use this command:

esxcfg-vswitch -m <MTU size> <vSwitch Name>

To delete a vSwitch, use this command:

esxcfg-vswitch -d <vSwitch Name>

Creating, Configuring, and Deleting Port Groups

As with virtual switches, the esxcfg-vswitch is the command you will use to work with port groups. Once again, unless I specifically indicate otherwise, all the commands, parameters, and arguments are case-sensitive.

To create a port group, use this command:

esxcfg-vswitch -A <Port Group Name> <vSwitch Name>

To set the VLAN ID for a port group, use this command:

esxcfg-vswitch -v <VLAN ID> -p <Port Group Name> <vSwitch Name>

To delete a port group, use this command:

esxcfg-vswitch -D <Port Group Name> <vSwitch Name>

To view the current list of vSwitches, port groups, and uplinks, use this command:

esxcfg-vswitch -l (lowercase L)

There are more networking-related tasks that you can perform from the CLI, but for a new user these commands should handle the lion’s share of all the networking configuration. Good luck!

Tags: , , , ,

In early April when I traveled to Houston to meet with HP about the ProLiant G6 servers—a trip which resulted in this blog post—I got into a discussion with representatives from HP and Intel regarding the Xeon 5500 CPUs and Enhanced VMotion Compatibility (EVC).

As you probably already know, EVC builds upon hardware extensions provided by AMD and Intel (AMD in the AMD-V extensions, Intel in FlexMigration) that allows a CPU to “alter” its personality or identification so as to allow live migration (i.e., VMotion) between CPUs that would otherwise be incompatible. Without these hardware extensions, you would have had to resort to unsupported CPU masking, which I’ve also discussed (here’s a summary of articles I’ve written on the topic). With the hardware extensions and EVC, the hypervisor can now take over the process of creating custom CPU masks that take all CPUs in the cluster down to the “lowest common denominator” (i.e., EVC masks off all features except those that are common to all CPUs in the cluster).

Ed Haletky provides a good overview of VMware EVC, in case you need more information.

Anyway, that’s enough background information. When I was in Houston, HP and Intel were discussing all the great new features that were available in the Xeon 5500. Knowing that introducing new CPU features generally meant introducing VMotion boundaries (like I describe here), I asked this question: “What happens to all these new features when you put a Xeon 5500-based server into a cluster with older servers and use EVC to guarantee VMotion compatibility?” My thought was that EVC would downgrade the Xeon 5500 by masking new features, thus negating many of the performance benefits offered by the Xeon 5500.

Obviously, the architectural changes introduced by the Xeon 5500 (specifically, Quick Path Interconnect) wouldn’t be affected, but what about Extended Page Tables (EPT)? Or VMDq? Or FlexPriority?

There were folks there from HP and Intel, and they were all stumped. They had to call in a few experts, but finally we got an answer from an Intel engineer on the VMware Alliance Team who was able to provide more information:

… The only features that would be disabled are some new instructions that are accessible to end-user applications living in Ring 3. For NHM-EP those would be the SSE4.2 and SSE4.1 instructions… No other features would be compromised. All the features of NHM-EP that provide performance and are under the direct control of the OS or hypervisor in Ring 0 will be fully usable: EPT, HT, NuMA, FlexPriority, VMDq, VT-x, VT-d, etc.

So there you have it: you can mix Xeon 5500-based servers in with older servers running previous generations of Xeon CPUs without sacrificing the performance benefits gained from the Xeon 5500.

Tags: , , , , ,

Upgrading a VMware Infrastructure 3.x environment to VMware vSphere 4 involves more than just upgrading vCenter Server and upgrading your ESX/ESXi hosts (as if that wasn’t enough). You should also plan on upgrading your virtual machines. VMware vSphere introduces a new hardware version (version 7), and vSphere also introduces a new paravirtualized network driver (VMXNET3) as well as a new paravirtualized SCSI driver (PVSCSI). To take advantage of these new drivers as well as other new features, you’ll need to upgrade your virtual machines. This process I describe below works really well.

I’d like to thank Erik Bussink, whose posts on Twitter got me started down this path.

Please note that this process will require some downtime. I personally tested this process with both Windows Server 2003 R2 as well as Windows Server 2008; it worked flawlessly with both versions of Windows. (I’ll post a separate article on doing something similar with other operating systems, if it’s even possible.)

  1. Record the current IP configuration of the guest operating system. You’ll end up needing to recreate it.
  2. Upgrade VMware Tools in the guest operating system. You can do this by right-clicking on the virtual machine and selecting Guest > Install/Upgrade VMware Tools. When prompted, choose to perform an automatic tools upgrade. When the VMware Tools upgrade is complete, the virtual machine will reboot.
  3. After the guest operating system reboots and is back up again, shutdown the guest operating system. You can do this by right-clicking on the virtual machine and selecting Power > Shutdown Guest.
  4. Upgrade the virtual machine hardware by right-clicking the virtual machine and selecting Upgrade Virtual Hardware.
  5. In the virtual machine properties, add a new network adapter of the type VMXNET3 and attach it to the same port group/dvPort group as the first network adapter.
  6. Remove the first/original network adapter.
  7. Add a new virtual hard disk to the virtual machine. Be sure to attach it to SCSI node 1:x; this will add a second SCSI adapter to the virtual machine. The size of the virtual hard disk is irrelevant.
  8. Change the type of the newly-added second SCSI adapter to VMware Paravirtual.
  9. Click OK to commit the changes you’ve made to the virtual machine.
  10. Power on the virtual machine. When the guest operating system is fully booted, log in and recreate the network configuration you recorded for the guest back in step 1. Windows may report an error that the network configuration is already used by a different adapter, but proceed anyway. Once you’ve finished, shut down the guest operating system again.
  11. Edit the virtual machine to remove the second hard disk you just added.
  12. While still in the virtual machine properties, change the type of the original SCSI controller to VMware Paravirtual (NOTE: See update below.)
  13. Power on the virtual machine. When the guest operating system is fully booted up, log in.
  14. Create a new system environment variable named DEVMGR_SHOW_NONPRESENT_DEVICES and set the value to 1.
  15. Launch Device Manager and from the View menu select Show Hidden Devices.
  16. Remove the drivers for the old network adapter and old SCSI adapter. Close Device Manager and you’re done!

If you perform these steps on a template, then you can be assured that all future virtual machines cloned from this template also have the latest paravirtualized drivers installed for maximum performance.

Post any questions or clarifications in the comments. Thanks!

UPDATE: Per this VMware KB article, VMware doesn’t support using the PVSCSI adapter for boot devices. That is not to say that it doesn’t work (it does work), but that it is not supported. Thanks to Eddy for pointing that out in the comments!

Tags: , , , ,

In April 2008, I wrote an article on how to use jumbo frames with VMware ESX and IP-based storage (NFS or iSCSI). It’s been a pretty popular post, ranking right up there with the ever-popular article on VMware ESX, NIC teaming, and VLAN trunks.

Since I started working with VMware vSphere (now officially available as of 5/21/2009), I have been evaluating how to replicate the same sort of setup using ESX/ESXi 4.0. For the most part, the configuration of VMkernel ports to use jumbo frames on ESX/ESXi 4.0 is much the same as with previous versions of ESX and ESXi, with one significant exception: the vNetwork Distributed Switch (vDS, what I’ll call a dvSwitch). After a fair amount of testing, I’m pleased to present some instructions on how to configure VMkernel ports for jumbo frames on a dvSwitch.

How I Tested

The lab configuration for this testing was pretty straightforward:

  • For the physical server hardware, I used a group of HP ProLiant DL385 G2 servers with dual-core AMD Opteron processors and a quad-port PCIe Intel Gigabit Ethernet NIC.
  • All the HP ProLiant DL385 G2 servers were running the GA builds of ESX 4.0, managed by a separate physical server running the GA build of vCenter Server.
  • The ESX servers participated in a DRS/HA cluster and a single dvSwitch. The dvSwitch was configured for 4 uplinks. All other settings on the dvSwitch were left at the defaults.
  • For the physical switch infrastructure, I used a Cisco Catalyst 3560G running Cisco IOS version 12.2(25)SEB4.
  • For the storage system, I used an older NetApp FAS940. The FAS940 was running Data ONTAP 7.2.4.

Keep in mind that these procedures or commands may be different in your environment, so plan accordingly.

Physical Network Configuration

Refer back to my first article on jumbo frames to review the Cisco IOS commands for configuring the physical switch to support jumbo frames. Once the physical switch is ready to support jumbo frames, you can proceed with configuring the virtual environment.

Virtual Network Configuration

The virtual network configuration consists of several steps. First, you must configure the dvSwitch to support jumbo frames by increasing the MTU. Second, you must create a distributed virtual port group (dvPort group) on the dvSwitch. Finally, you must create the VMkernel ports with the correct MTU. Each of these steps is explained in more detail below.

Setting the MTU on the dvSwitch

Setting the MTU on the dvSwitch is pretty straightforward:

  1. In the vSphere Client, navigate to the Networking inventory view (select View > Inventory > Networking from the menu).
  2. Right-click on the dvSwitch and select Edit Settings.
  3. From the Properties tab, select Advanced.
  4. Set the MTU to 9000.
  5. Click OK.

That’s it! Now, if only the rest of the process was this easy…

By the way, this same area is also where you can enable Cisco Discovery Protocol support for the dvSwitch, as I pointed out in this recent article.

Creating the dvPort Group

Like setting the MTU on the dvSwitch, this process is pretty straightforward and easily accomplished using the vSphere Client:

  1. In the vSphere Client, navigate to the Networking inventory view (select View > Inventory > Networking from the menu).
  2. Right-click on the dvSwitch and select New Port Group.
  3. Set the name of the new dvPort group.
  4. Set the number of ports for the new dvPort group.
  5. In the vast majority of instances, you’ll want to set VLAN Type to VLAN and then set the VLAN ID accordingly. (This is the same as setting the VLAN ID for a port group on a vSwitch.)
  6. Click Next.
  7. Click Finish.

See? I told you it was pretty straightforward. Now on to the final step which, unfortunately, won’t be quite so straightforward or easy.

Creating a VMkernel Port With Jumbo Frames

Now things get a bit more interesting. As of the GA code, the vSphere Client UI still does not expose an MTU setting for VMkernel ports, so we are still relegated to using the esxcfg-vswitch command (or the vicfg-vswitch command in the vSphere Management Assistant—or vMA—if you are using ESXi). The wrinkle comes in the fact that we want to create a VMkernel port attached to a dvPort ID, which is a bit more complicated than simply creating a VMkernel port attached to a local vSwitch.

Disclaimer: There may be an easier way than the process I describe here. If there is, please feel free to post it in the comments or shoot me an e-mail.

First, you’ll need to prepare yourself. Open the vSphere Client and navigate to the Hosts and Clusters inventory view. At the same time, open an SSH session to one of the hosts you’ll be configuring, and use “su -” to assume root privileges. (You’re not logging in remotely as root, are you?) If you are using ESXi, then obviously you’d want to open a session to your vMA and be prepared to run the commands there. I’ll assume you’re working with ESX.

This is a two-step process. You’ll need to repeat this process for each VMkernel port that you want to create with jumbo frame support.

Here are the steps to create a jumbo frames-enabled VMkernel port:

  1. Select the host and and go the Configuration tab.
  2. Select Networking and change the view to Distributed Virtual Switch.
  3. Click the Manage Virtual Adapters link.
  4. In the Manage Virtual Adapters dialog box, click the Add link.
  5. Select New Virtual Adapter, then click Next.
  6. Select VMkernel, then click Next.
  7. Select the appropriate port group, then click Next.
  8. Provide the appropriate IP addressing information and click Next when you are finished.
  9. Click Finish. This returns you to the Manage Virtual Adapters dialog box.

From this point on you’ll go the rest of the way from the command line. However, leave the Manage Virtual Adapters dialog box open and the vSphere Client running.

To finish the process from the command line:

  1. Type the following command (that’s a lowercase L) to show the current virtual switching configuration:
    esxcfg-vswitch -l
    At the bottom of the listing you will see the dvPort IDs listed. Make a note of the dvPort ID for the VMkernel port you just created using the vSphere Client. It will be a larger number, like 266 or 139.
  2. Delete the VMkernel port you just created:
    esxcfg-vmknic -d -s <dvSwitch Name> -v <dvPort ID>
  3. Recreate the VMkernel port and attach it to the very same dvPort ID:
    esxcfg-vmknic -a -i <IP addr> -n <Mask> -m 9000 -s <dvSwitch Name> -v <dvPort ID>
  4. Use the esxcfg-vswitch command again to verify that a new VMkernel port has been created and attached to the same dvPort ID as the original VMkernel port.

At this point, you can go back into the vSphere Client and enable the VMkernel port for VMotion or FT logging. I’ve tested jumbo frames using VMotion and everything is fine; I haven’t tested FT logging with jumbo frames as I don’t have FT-compatible CPUs. (Anyone care to donate some?)

As I mentioned in yesterday’s Twitter post, I haven’t conducted any objective performance tests yet, so don’t ask. I can say that NFS feels faster with jumbo frames than without, but that’s purely subjective.

Let me know if you have any questions or if anyone finds a faster or easier way to accomplish this task.

UPDATE: I’ve updated the comments to delete and recreate the VMkernel port per the comments below.

Tags: , , , , , , , , ,

So here’s another “thinking out loud” post. This time, I’m thinking about Fibre Channel over Ethernet (FCoE) and unified fabric.

I was going back through a list of blog posts and articles that I wanted to read and think on, and I came across a link to Dave Graham’s article titled Moving a Fabric forward: FCoE Adoption and other Questions. His blog entry was partially in response to my FCoE discussion post. His post got me thinking again.

It seems like anytime someone talks about FCoE, they end up also talking about unified fabric. After having read a number of different articles and posts regarding FCoE, I can see where FCoE would be attractive to shops with significant FCP installations. In my mind, though, this doesn’t necessarily mean unified fabric. Given the political differences in organizations—think the “storage team” and the “networking team”—how likely is it that an organization may adopt FCoE, but not unified fabric? Or how likely is it that an organization may adopt FCoE, intending it to be a transitional technology leading to unified fabric, but never actually make it all the way?

So here’s my question: is unified fabric an inevitability?

(Oh, and here’s a related question: Most people cite VoIP as proof that the unified fabric is inevitable. More so than anything else, I believe VoIP’s success was more a reflection of the rising importance of TCP/IP networking. If so, does that give iSCSI an edge over FCoE? Is iSCSI the “VoIP of the storage world”?)

Tags: , , ,

One of my projects involved the configuration of GRE (Generic Routing Encapsulation) tunnels, encrypted by IPSec, between two locations. I was having some problems getting the tunnels to work properly, but now I’ve managed to resolve that problem, and the configuration is working well. Here’s some additional information on the problem and how it was finally corrected.

This was my first project using GRE tunnels. I’d used IPSec tunnels many times, and on many different platforms, but this time around we needed an interface that could be tracked for HSRP (Hot Standby Router Protocol) purposes, and until recently Cisco didn’t offer IPSec tunnel interfaces. (I just came across some documentation last night that indicated very recent releases of IOS offer this functionality.) So, the idea was to use GRE tunnels, track the GRE tunnels using HSRP for failover with another router, and encrypt the traffic using IPSec in transport mode.

The GRE tunnel configuration (scrubbed for sensitive data) looked something like this originally:

interface Tunnel0
 description GRE tunnel to other location
 ip address 192.168.254.1 255.255.255.252
 tunnel source FastEthernet0/0
 tunnel destination 172.31.254.1
 crypto map tunnel-ipsec-map

Of course, there was an appropriately configured interface at the other end of the tunnel as well. The tunnels came up, and appeared to work just fine, until we added the keepalive statement. (The keepalive statement is required for the tunnel to report an actual up/down status, necessary for HSRP interface tracking.) Then they went down and stayed down.

A “debug tunnel” statement showed that the keepalives were being sent, but none were being received. Thinking perhaps the IPSec configuration was incorrect, I removed the “crypto map” statement from the tunnel interface. It still didn’t work.

After reviewing the configuration again, I began to suspect an MTU issue—the “show int tun0″ output listed an MTU of 1514. I consulted with a Cisco expert (recently obtained his CCIE), and he confirmed that it was most likely an MTU issue. So I modified the configuration to look like this:

interface Tunnel0
 description GRE tunnel to other location
 ip address 192.168.254.1 255.255.255.252
 ip mtu 1400
 keepalive
 tunnel source FastEthernet0/0
 tunnel destination 172.31.254.1

At that point, the tunnel finally came up and I was able to pass traffic through the tunnel. I re-added the “crypto map” statement to enforce encryption, and the tunnels promptly went back down again.

Once again, debug output saved the day. The output from a “debug crypto” statement was constantly reporting “packet too small”. A search of the Cisco web site turned up a result (I can’t find it now) that indicated a bug within IOS and suggested the addition of a “tunnel key” statement. So, I modified the configuration again:

interface Tunnel0
 description GRE tunnel to other location
 ip address 192.168.254.1 255.255.255.252
 ip mtu 1400
 keepalive
 tunnel source FastEthernet0/0
 tunnel destination 172.31.254.1
 tunnel key 12345
 crypto map tunnel-ipsec-map

With this configuration, the IPSec/ISAKMP SAs were established and the tunnels came up, passing traffic as expected. The debug output showed no crypto errors, and keepalives were being sent and received. Success!

Tags: , , , ,

Newer entries »