I haven’t hid the fact that I’m not really a fan of FCoE (see here or here), but I was starting to warm to the technology and thought that I was beginning to see some benefits to deploying FCoE. Namely, the fact that FCoE is inherently very compatible with “traditional” FCP, allowing organizations to leverage their existing FCP installation while transitioning to FCoE. Some hands-on time I’d recently spent with a Cisco Nexus 5000 switch showed me just how closely aligned the two technologies are and how (relatively) easy it was to extend an FC fabric using FCoE. OK, I think I get this.

Then, a few days ago, I read this article on FCoE divergence. Given that The Register can sometimes be quite sensationalist (and that’s putting it mildly), I contacted a colleague of mine whose input and knowledge I trust. He informed me that FCoE was currently limited in that FCoE is not multi-hop enabled—meaning, you can’t connect FCoE initiators on one switch to FCoE targets on another switch. (Apparently, this shortcoming is due to be corrected shortly.)

Whoa! That’s a limitation of which I was not aware. And with that limitation in mind, knowing that FCoE will—for the time being at least—be limited to convergence at the edge, I have to ask: why deploy FCoE at all? What real and specific benefits does an organization seek to gain by deploying FCoE as opposed to just deploying FC? Is the edge convergence really that worthwhile and valuable?

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

Thinking Out Loud: Why Deploy FCoE?

• Is Unified Fabric an Inevitability?
• A Collection of Viewpoints on Cisco UCS
• FCoE versus MR-IOV…huh?
• Continuing the FCoE Discussion
• Blades Won’t Die, But They Will Change

I do agree with Steve that there is no innate relationship that means running VMware on blades is somehow “automagically” better:

It is amazing how frequently we hear IT managers talk about deploying blade servers as an integral component of their new virtual infrastructures - as if there were an obvious synergy between VMware and blade server architectures.

Absolutely! Blades are an option, just like rack mounted servers, and it’s up to the customer to choose (or us as consultants to recommend) the form factor that best meets the business needs. It might be blade servers, or it might be rack mounted servers. It just depends. So, on this one point, I agree with Steve.

Yet, at the same time, I also disagree with this point that Steve makes in his article:

Blade servers have always been an impediment to an optimal virtual infrastructure because they introduce limitations in efficiently utilizing power and cooling resources, budget, flexibility, manageability, bios and firmware updates, performance and troubleshooting.

Here is where Steve and I start to disagree. In fact, this specific article was something of the catalyst for a series of posts, written by colleague and friend Aaron Delp, detailing how blade servers and virtualization work well together:

Blades and Virtualization Aren’t Mutually Exclusive: Part One, HP Power Sizing
Blades and Virtualization Aren’t Mutually Exclusive: Part Two, IBM Power Sizing
Blades and Virtualization Aren’t Mutually Exclusive: Part Three, IBM Traditional Expansion Options
Blades and Virtualization Aren’t Mutually Exclusive: Part Four, HP Traditional Expansion Options

While this series of articles doesn’t squarely address all of the arguments against blades and virtualization, the series does make it clear that blades can produce power savings vs. rack mounted servers, and that blades do offer enough expansion options to accommodate the majority of virtualization deployments.

I also disagree with Steve about the value of the unified fabric, especially considering that right now unified fabric can exist only at the edge of the network and not at the core. That being the case, I find it hard to say that unified fabric is going to kill blade servers. So, again, I have to disagree with Steve’s position.

However, Steve’s not entirely wrong—virtualization, FCoE and 10Gb Ethernet, and yes even unified fabric will change how blade servers are designed and deployed. Cisco’s Unified Computing System (UCS) is one example of how blade servers are going to adapt to these agents of change, and I believe we’ll see more examples from other leading vendors in the coming months and years. But will blades die away entirely? No, I don’t think so.

Think I’m crazy? Think I’m out of my mind? Feel free to speak up in the comments—courteous comments are always welcome.

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

Blades Won’t Die, But They Will Change

• Alas, BladeVault is No More
• Blades and Virtualization Aren’t Mutually Exclusive: Part Three, IBM Traditional Expansion Options
• Blades and Virtualization Aren’t Mutually Exclusive: Part Two, IBM Power Sizing
• Blades and Virtualization Aren’t Mutually Exclusive: Part One, HP Power Sizing
• Blades and Virtualization Aren’t Mutually Exclusive: Part Four, HP Traditional Expansion Options

For new users who are seeking a thorough explanation of how VMware ESX networking functions, I’ll recommend a series of articles by Ken Cline titled The Great vSwitch Debate. Ken goes into a great level of detail. Go read that, then you can come back here.

Before I get started it’s important to understand that, for the most part, the information in this article applies only to VMware ESX. VMware ESXi doesn’t have a Linux-based Service Console like VMware ESX, and therefore doesn’t have a readily-accessible CLI from which to run these sorts of commands. There is a remote CLI available, which I’ll discuss in a future post, but for now I’ll focus only on VMware ESX.

The majority of all the networking configuration you will need to perform on VMware ESX boils down to just a couple commands:

• esxcfg-vswitch: You will use this command to manipulate virtual switches (vSwitches) and port groups.
• esxcfg-nics: You will use this command to view (and potentially manipulate) the physical network interface cards (NICs) in the VMware ESX host.

Configuring VMware ESX networking boils down to a couple basic tasks:

1. Creating, configuring, and deleting vSwitches
2. Creating, configuring, and deleting port groups

I’ll start with creating, configuring, and deleting vSwitches.

Creating, Configuring, and Deleting vSwitches

You’ll primarily use the esxcfg-vswitch command for the majority of these tasks. Unless I specifically indicate otherwise, all the commands, parameters, and arguments are case-sensitive.

To create a vSwitch, use this command:

esxcfg-vswitch -a <vSwitch Name>

To link a physical NIC to a vSwitch—which is necessary in order for the vSwitch to pass traffic onto the physical network or to receive traffic from the physical network—use this command:

esxcfg-vswitch -L <Physical NIC> <vSwitch Name>

In the event you don’t have information on the physical NICs, you can use this command to list the physical NICs:

esxcfg-nics -l (lowercase L)

Conversely, if you need to unlink (remove) a physical NIC from a vSwitch, use this command:

esxcfg-vswitch -U <Physical NIC> <vSwitch Name>

To change the Maximum Transmission Unit (MTU) size on a vSwitch, use this command:

esxcfg-vswitch -m <MTU size> <vSwitch Name>

To delete a vSwitch, use this command:

esxcfg-vswitch -d <vSwitch Name>

Creating, Configuring, and Deleting Port Groups

As with virtual switches, the esxcfg-vswitch is the command you will use to work with port groups. Once again, unless I specifically indicate otherwise, all the commands, parameters, and arguments are case-sensitive.

To create a port group, use this command:

esxcfg-vswitch -A <Port Group Name> <vSwitch Name>

To set the VLAN ID for a port group, use this command:

esxcfg-vswitch -v <VLAN ID> -p <Port Group Name> <vSwitch Name>

To delete a port group, use this command:

esxcfg-vswitch -D <Port Group Name> <vSwitch Name>

To view the current list of vSwitches, port groups, and uplinks, use this command:

esxcfg-vswitch -l (lowercase L)

There are more networking-related tasks that you can perform from the CLI, but for a new user these commands should handle the lion’s share of all the networking configuration. Good luck!

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

New User’s Guide to Configuring VMware ESX Networking via CLI

• Identifying ESX Server NICs in Blades
• ESX Server, IP Storage, and Jumbo Frames
• Setting VMware ESX vSwitch Load Balancing Policy via CLI
• Using vmware-vim-cmd to Modify a Portgroup
• ESX Server, NIC Teaming, and VLAN Trunking

• Trigged by a series of blog posts by Arnim van Lieshout on VMware ESX memory management (Part 1, Part 2, and Part 3), Scott Herold decided to join the fray with this blog post. Both Scott’s post and Arnim’s posts are good reading for anyone interested in getting a better idea of what’s happening “under the covers,” so to speak, when it comes to memory management.
• Perhaps prompted by my post on upgrading virtual machines in vSphere, a lot of information has come to light regarding the PVSCSI driver. Some are advocating changes to best practices to incorporate the PVSCSI driver, but others seem to be questioning the need to move away from a single drive model (a necessary move since PVSCSI isn’t supported for boot drives). Personally, I just want VMware to support the PVSCSI driver on boot drives.
• Eric Sloof confirms for us that name resolution is still the Achilles’ Heel of VMware High Availability in VMware vSphere.
• I don’t remember where I picked up this VMware KB article, but it sure would be handy if VMware could provide more information about the issue, such as what CPUs might be affected. Otherwise, you’re kind of shooting in the dark, aren’t you?
• Upgraded to VMware vSphere, and now having issues with VMotion? Thanks to VMwarewolf, this pair of VMware KB articles (here and here) might help resolve the issue.
• Chad Sakac of EMC and co-conspirator for the storage portion of Mastering VMware vSphere 4 (pre-order here), has been putting out some very good posts:

• More on Exchange on vSphere (including FT)
• Integrated vSphere enterprise workloads - all together, at scale
• Using vSphere and HW offload for improved Celerra VSA performance
• vSphere and 2TB LUNs - changes from VI3.x

OK, I guess that should just about do it. Thanks for reading, and please share your thoughts, interesting links, or (pertinent) rants in the comments.

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

The Return of Virtualization Short Takes

• Adaptec Support in vSphere Revoked?
• I Love Having My Content Stolen
• Storage Short Take #2
• VMware vSphere Generates Insane Amounts of I/O
• vSphere Virtual Machine Upgrade Process

My wife made an offhand comment that she couldn’t access the shared music library from her laptop. I tested the connection and, sure enough, every time I clicked the shared library icon it simply disappeared. No error, no warning, no entries in any log files…it just disappeared. I searched the Windows event logs, and I searched the log files on the Ubuntu server downstairs. Neither computer had any entries whatsoever that provided any insight as to why this one computer would not connect to the shared music library.

Being the geeky troubleshooter that I am, I attempted to replicate the problem on some of the other computers on the network. My MacBook Pro worked fine. Three other Windows laptops on the network, running the same version of Windows (Windows XP Professional) and the same Service Pack revision, also worked fine. The problem seemed to be isolated to her computer. Perhaps it was only when she was on the wireless network…nope, the same problem regardless of the network connection.

I upgraded iTunes to the latest version. That didn’t work. I disabled the Windows Firewall on her computer. That didn’t work. I made sure that no traffic was being blocked by the firewall on the Ubuntu server; no traffic was being blocked. In other words, that didn’t work. I was about to give up and just write it off as one of those strange aberrations that couldn’t be resolved and chalk it up to Windows.

Then I stumbled onto this site. I’d already created a daapd.service file for Avahi to use previously, but this site described some additional entries in the daapd.service file that I didn’t have. I made some edits, based on the information on the site, and here’s the daapd.service file I had for Avahi:

<?xml version="1.0" standalone='no'?><!--*-nxml-*-->
<!DOCTYPE service-group SYSTEM “avahi-service.dtd”>
<service-group>
<name replace-wildcards=”yes”>Home Media Server</name>
<service>
<type>_daap._tcp</type>
<port>3689</port>
<txt-record>txtvers=1</txt-record>
<txt-record>iTSh Version=131073</txt-record>
<txt-record>Version=196610</txt-record>
</service>
</service-group>

After changing the daapd.service file to the version listed above, I restarted Avahi. Upon the shared media server re-appearing in iTunes, I clicked on it and…drum roll please…it worked! The previous version I had been using did not have the txt-record entries, and I really have no idea why adding the txt-record entries suddenly made my wife’s iTunes connect properly. I suppose it doesn’t matter why it works, it just matters that I FIXED IT! (ePlus engineers who attended our NSM this year will get this joke.)

Still, in the event you’re running into the same issue—a Windows installation of iTunes that fails to connect to a shared music library running on Firefly Media Server—then perhaps updating your Avahi configuration will correct the problem.

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

Strange Error with Firefly Media Server, iTunes, and Avahi

• OpenVPN and mt-daapd
• Ubuntu and Mac OS X Integration
• My Ubuntu-Mac Article is Popular, I Guess
• Some New Christian Music
• The Power of Quicksilver

Since I started working with VMware vSphere (now officially available as of 5/21/2009), I have been evaluating how to replicate the same sort of setup using ESX/ESXi 4.0. For the most part, the configuration of VMkernel ports to use jumbo frames on ESX/ESXi 4.0 is much the same as with previous versions of ESX and ESXi, with one significant exception: the vNetwork Distributed Switch (vDS, what I’ll call a dvSwitch). After a fair amount of testing, I’m pleased to present some instructions on how to configure VMkernel ports for jumbo frames on a dvSwitch.

How I Tested

The lab configuration for this testing was pretty straightforward:

• For the physical server hardware, I used a group of HP ProLiant DL385 G2 servers with dual-core AMD Opteron processors and a quad-port PCIe Intel Gigabit Ethernet NIC.
• All the HP ProLiant DL385 G2 servers were running the GA builds of ESX 4.0, managed by a separate physical server running the GA build of vCenter Server.
• The ESX servers participated in a DRS/HA cluster and a single dvSwitch. The dvSwitch was configured for 4 uplinks. All other settings on the dvSwitch were left at the defaults.
• For the physical switch infrastructure, I used a Cisco Catalyst 3560G running Cisco IOS version 12.2(25)SEB4.
• For the storage system, I used an older NetApp FAS940. The FAS940 was running Data ONTAP 7.2.4.

Keep in mind that these procedures or commands may be different in your environment, so plan accordingly.

Physical Network Configuration

Refer back to my first article on jumbo frames to review the Cisco IOS commands for configuring the physical switch to support jumbo frames. Once the physical switch is ready to support jumbo frames, you can proceed with configuring the virtual environment.

Virtual Network Configuration

The virtual network configuration consists of several steps. First, you must configure the dvSwitch to support jumbo frames by increasing the MTU. Second, you must create a distributed virtual port group (dvPort group) on the dvSwitch. Finally, you must create the VMkernel ports with the correct MTU. Each of these steps is explained in more detail below.

Setting the MTU on the dvSwitch

Setting the MTU on the dvSwitch is pretty straightforward:

1. In the vSphere Client, navigate to the Networking inventory view (select View > Inventory > Networking from the menu).
2. Right-click on the dvSwitch and select Edit Settings.
3. From the Properties tab, select Advanced.
4. Set the MTU to 9000.
5. Click OK.

That’s it! Now, if only the rest of the process was this easy…

By the way, this same area is also where you can enable Cisco Discovery Protocol support for the dvSwitch, as I pointed out in this recent article.

Creating the dvPort Group

Like setting the MTU on the dvSwitch, this process is pretty straightforward and easily accomplished using the vSphere Client:

1. In the vSphere Client, navigate to the Networking inventory view (select View > Inventory > Networking from the menu).
2. Right-click on the dvSwitch and select New Port Group.
3. Set the name of the new dvPort group.
5. Set the number of ports for the new dvPort group.
6. In the vast majority of instances, you’ll want to set VLAN Type to VLAN and then set the VLAN ID accordingly. (This is the same as setting the VLAN ID for a port group on a vSwitch.)
7. Click Next.
8. Click Finish.

See? I told you it was pretty straightforward. Now on to the final step which, unfortunately, won’t be quite so straightforward or easy.

Creating a VMkernel Port With Jumbo Frames

Now things get a bit more interesting. As of the GA code, the vSphere Client UI still does not expose an MTU setting for VMkernel ports, so we are still relegated to using the esxcfg-vswitch command (or the vicfg-vswitch command in the vSphere Management Assistant—or vMA—if you are using ESXi). The wrinkle comes in the fact that we want to create a VMkernel port attached to a dvPort ID, which is a bit more complicated than simply creating a VMkernel port attached to a local vSwitch.

Disclaimer: There may be an easier way than the process I describe here. If there is, please feel free to post it in the comments or shoot me an e-mail.

First, you’ll need to prepare yourself. Open the vSphere Client and navigate to the Hosts and Clusters inventory view. At the same time, open an SSH session to one of the hosts you’ll be configuring, and use “su -” to assume root privileges. (You’re not logging in remotely as root, are you?) If you are using ESXi, then obviously you’d want to open a session to your vMA and be prepared to run the commands there. I’ll assume you’re working with ESX.

This is a two-step process. You’ll need to repeat this process for each VMkernel port that you want to create with jumbo frame support.

Here are the steps to create a jumbo frames-enabled VMkernel port:

1. Select the host and and go the Configuration tab.
2. Select Networking and change the view to Distributed Virtual Switch.
3. Click the Manage Virtual Adapters link.
4. In the Manage Virtual Adapters dialog box, click the Add link.
5. Select New Virtual Adapter, then click Next.
6. Select VMkernel, then click Next.
7. Select the appropriate port group, then click Next.
8. Provide the appropriate IP addressing information and click Next when you are finished.
9. Click Finish. This returns you to the Manage Virtual Adapters dialog box.

From this point on you’ll go the rest of the way from the command line. However, leave the Manage Virtual Adapters dialog box open and the vSphere Client running.

To finish the process from the command line:

1. Type the following command (that’s a lowercase L) to show the current virtual switching configuration:
   esxcfg-vswitch -l
   At the bottom of the listing you will see the dvPort IDs listed. Make a note of the dvPort ID for the VMkernel port you just created using the vSphere Client. It will be a larger number, like 266 or 139.
2. Delete the VMkernel port you just created:
   esxcfg-vmknic -d <dvPort ID>
3. Recreate the VMkernel port and attach it to the very same dvPort ID:
   esxcfg-vmknic -a -i <IP addr> -n <Mask> -m 9000 <dvPort ID>
4. Use the esxcfg-vswitch command again to verify that a new VMkernel port has been created and attached to the same dvPort ID as the original VMkernel port.

At this point, you can go back into the vSphere Client and enable the VMkernel port for VMotion or FT logging. I’ve tested jumbo frames using VMotion and everything is fine; I haven’t tested FT logging with jumbo frames as I don’t have FT-compatible CPUs. (Anyone care to donate some?)

As I mentioned in yesterday’s Twitter post, I haven’t conducted any objective performance tests yet, so don’t ask. I can say that NFS feels faster with jumbo frames than without, but that’s purely subjective.

Let me know if you have any questions or if anyone finds a faster or easier way to accomplish this task.

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

VMware vSphere vDS, VMkernel Ports, and Jumbo Frames

• ESX Server, IP Storage, and Jumbo Frames
• VLANs and Port Groups
• New User’s Guide to Configuring VMware ESX Networking via CLI
• Xsigo I/O Director Tips and Tricks
• vSphere Virtual Machine Upgrade Process

I’ll start with Robin Harris aka StorageMojo and his initial take on UCS. The one thing that jumped out at me about his article was this statement:

If IBM, HP and Sun aren’t meeting today to plot a radical, Cisco margin destroying open-source router & low-cost switch counterattack - like Seagate, HP and IBM performed on Quantum’s DLT - they’re idiots.

This seems to validate the strategy outlined by Sun and sheds new light—for me, at least—on the potential motivations for IBM to acquire Sun and, thus, Sun’s intellectual property. Is Big Blue’s move to acquire Sun a precursor to a strike against the heart of Cisco’s routing and switching business? And how would Cisco respond to just such a move?

Massimo Re Ferre’ of IT 2.0 approaches UCS from a different angle. According to Massimo, if you stop and really look at what you get from UCS, it’s not terribly different from what you can get from other vendors. In fact, if you separate out the unified fabric, there really isn’t a whole lot to distinguish UCS from other, similar solutions from HP, IBM, Dell, and Sun. And if you think about it, he’s right—it’s really only the unified fabric, along with the fabric extenders in the chassis and the single point of management, that differentiate the platform.

Therein lies the problem. Massimo points out a couple of potential problems with unified fabric (security and political/organizational challenges). If unified fabric doesn’t fly, then UCS is grounded too. And industry excitement over FCoE isn’t exactly the greatest in the world. Chris Evans aka The Storage Architect makes clear his feelings about FCoE in this post:

FCoE is a Cisco strategy to own the data centre, nothing else. As the recession bites, it would be a brave soul who could justify the disruption and additional spend, for very little gain.

FCoE is hardly a forgone conclusion, and given that so much of UCS’ value is tied up in the unified fabric and the results that come from it, that makes UCS awfully vulnerable.

Not everyone thinks that FCoE makes the UCS vulnerable, by the way; technology evangelist Christopher Kusek thinks the future will be a unified one:

The Data center has spoken and it’s answer is True unification.

Burton Group analyst Drue Reeves says that this was a move Cisco had to make:

In the end, UCS was a move Cisco had to make to ward off competition AND increase shareholder value. Cisco has a strong brand, enterprise credibility, the technical chops and finances to pull it off. Is UCS a business risk? Sure. But the greater risk for Cisco is to do nothing.

Perhaps, perhaps not. Given that UCS relies so heavily on FCoE, wouldn’t it have made sense for Cisco to push the FCoE train along by providing FCoE interconnects for blade servers from HP and IBM? Of course, this is supposing that CNAs would be available for HP and IBM blades, but I’m sure that this is something Cisco could have helped guarantee. This route would have broadened the market for FCoE and the unified fabric and simultaneously establishing Cisco as the FCoE leader (as if they weren’t already). Then, when really game-changing stuff like the SR-IOV-enabled adapters like “Palo” were available, Cisco could have taken the leap into the compute space and played the unified management card. That seems like a less risky approach to me. But hey, what do I know?

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

A Collection of Viewpoints on Cisco UCS

• Is Unified Fabric an Inevitability?
• Cisco UCS Announcements
• Blades Won’t Die, But They Will Change
• More on Cisco UCS
• Continuing the FCoE Discussion

Viewing CDP data from the Cisco switch is easy. From the switch’s command prompt, use this command:

show cdp neighbors

This will display the CDP information that the switch has gathered. When CDP is enabled on ESX/ESXi, that will include information on which VMnics are attached to which switch ports.

From the ESX side, you can use this command:

esxcfg-info | more +/CDP\ Summary

This searches for the string “CDP Summary” in the output of the esxcfg-info command. The output from that command will include information about the switch to which the ESX host is connected, the ports to which the NICs are connected, and associated VLANs. The screenshot below shows some of the output from this command.

Thanks go to reader Larry for the information on this command. Other readers, feel free to continue to share information here. It is helpful!

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

Viewing CDP Data on VMware ESX

• Summary of CDP Articles
• Identifying ESX Server NICs in Blades
• VMware vSphere vDS, VMkernel Ports, and Jumbo Frames
• New User’s Guide to Configuring VMware ESX Networking via CLI
• LACP with Cisco Switches and NetApp VIFs

If you look back at the diagram that’s included in that UCS post, toward the bottom you’ll see a very small blurb about some Cisco UCS Network Adapters that are optimized for efficiency and performance, compatibility, and virtualization. In a nutshell, the idea here is that there are three different CNA families targeted at different markets: high-performance Ethernet, compatibility with existing driver stacks, and virtualization. Users will choose the CNA that best suits their needs. For the purposes of this post, I’d like to discuss the virtualization-optimized CNA.

The idea here is that the virtualization-optimized CNA (what is being referred to as “Palo”) will leverage a number of different technologies in virtualized environments:

• It will utilize SR-IOV (Single Root I/O Virtualization), a PCI SIG standard for allowing a physical network adapter to present multiple virtual adapters to upper-level software, in this case the hypervisor. This eliminates the need for the hypervisor to manage the physical network adapter and allows VMs to attach directly to one of the SR-IOV virtual adapters (or, as Brad Hedlund put it in this comment to my original article, an “SR-IOV slice of the adapter”).
• It will utilize Intel I/O Acceleration Technology (Intel I/OAT) to minimize bottlenecks in the hardware and allow the server to better cope with massive dataflows like those generated by 10GbE adapters.
• It will use Intel Virtual Machine Device Queues (VMDq) to improve traffic management within the server and decrease the processing burden on the VMM, i.e., the hypervisor.

Together, these technologies can be referred to as Intel VT-c. The virtualization-optimized drivers will also take advantage of Intel VT-d to provide hardware-assisted DMA remapping and better protection and performance of direct-assigned devices.

“OK,” you say. “But where is all this leading?” Good question! Let’s bring it all together.

Today, in the VMware space, virtual machines are connected to a vSwitch because connecting them directly to a physical adapter just isn’t practical. Yes, there is VMDirectPath, but for VMDirectPath to really work it needs more robust hardware support. Otherwise, you lose useful features like VMotion. (Refer back to my VMworld 2008 session notes from TA2644.) So, we have to manage physical switches and virtual switches—that’s two layers of management and two layers of switching. Along comes the Cisco Nexus 1000V. The 1000V helps to centralize management but we still have two layers of switching.

That’s where the “Palo” adapter comes in. Using VMDirectPath “Gen 2″ (again, refer to my TA2644 notes) and the various hardware technologies I listed and described above, we now gain the ability to attach VMs directly to the network adapter and eliminate the virtual switching layer entirely. Now we’ve both centralized the management and eliminated an entire layer of switching. And no matter how optimized the code may be, the fact that the hypervisor doesn’t have to handle packets means it has more cycles to do other things. In other words, there’s less hypervisor overhead. I think we can all agree that’s a good thing.

<aside>I’ve clashed with a couple of different people thus far because of differences on perspective with regard to UCS. OK, specifically, it’s been regarding people who insist that UCS isn’t a blade server. OK, UCS as an overall system is not a blade server, but the B-Series blades are a significant part of this overall system—so to say that Cisco’s isn’t building blade servers really isn’t accurate. They are building blade servers, but these are blade servers with an as-yet-unseen level of integration with other technologies. If there is one area in which UCS stands apart from any other blade server-related solution on the market, it would be this level of integration, especially the integration with virtualization technology.</aside>

Chad Sakac of EMC touches on this very lightly in his latest post. Being who I am, though, I much prefer digging a bit deeper to better understand exactly what’s going on.

UCS experts, feel free to correct me or clarify my statements in the comments. Thanks!

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

Cisco UCS Virtualization-Optimized CNAs

• More on Cisco UCS
• Technology Convergence
• A Collection of Viewpoints on Cisco UCS
• Cisco UCS Announcements
• TA2644: Networking I/O Virtualization

UCS essentially consists of 4 major components:

• The UCS 6100 Series Fabric Interconnect devices, running Cisco UCS Manager
• The UCS 2100 Series Fabric Extender, with up to 2 of them running in each chassis
• The UCS B-Series blades, either half-width (8 blades per chassis) or full-width (4 blades per chassis), and up to 40 chassis per system
• UCS network adapters supporting DCE/CEE/DCB and FCoE, apparently coming in three different flavors (efficiency/performance, compatibility, and virtualization)

This diagram shows an overview of UCS:

With the exception of the UCS Manager software and the Converged Network Adapters (CNAs), everything else is pretty standard stuff:

• The UCS 6100 is essentially a Nexus 5000, but with the ability to run the UCS Manager software.
• The UCS 2100 is essentially the same as the Nexus 2000 Fabric Extender (FEX), but in a form factor that is intended to plug into the UCS chassis.
• The B-Series blades are industry standard blades running Intel Nehalem CPUs, standard hot-plug hard drives, and 10Gb CNAs.

The CNAs appear to be one area in which there may be some innovation. In particular, the virtualization-optimized CNA appears to extend some new functionality into the virtualization layer itself, although it’s currently unclear exactly how, or how the virtualization layer will leverage that functionality. It sounds like SR-IOV to me, but others are indicating that it’s an offshoot of Intel’s VT-d technology. Speaking specifically with regard to VMware ESX/ESXi, I would guess that this will need to be combined with VMDirectPath, as it appears to replace the need for the vSwitch within the ESX/ESXi host. Personally, I’d rather not replace the vSwitch and instead allow the UCS 6100 and/or UCS Manager to manage Nexus 1000V VEMs on the ESX/ESXi hosts instead. This will truly bring unification without adding complexity.

The real wildcard here is UCS Manager. Although the Cisco webcast spoke frequently of the “open APIs” and “XML APIs” that other partners can leverage, but nothing substantial or significant was released regarding UCS Manager. Lots of questions have yet to be answered, but the one that really jumps out at me is this one:

How will an organization need to organize their storage in order to take advantage of UCS?

I’m guessing here that organizations will need to do boot-from-SAN in order to gain the true flexiblity and agility that UCS is supposed to provide. In that case, what Cisco is supplying is not that dramatically different from a multi-vendor solution that utilizes something like Scalent to provide automation. Of course, Cisco’s solution is from a single vendor and is supposedly more integrated.

So, there are my initial thoughts. What about you?

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

More on Cisco UCS

• Blades Won’t Die, But They Will Change
• A Collection of Viewpoints on Cisco UCS
• Next-Gen Stuff: Enabling CDP in ESX/ESXi
• Cisco UCS Virtualization-Optimized CNAs
• P2V With IBM Blades