Networking

It turns out that a couple more articles of mine have been published by SearchServerVirtualization.com, a sister site to SearchVMware.com where some of my other articles were also published.

These articles kind of snuck past me; I usually pick up on the publication of one of my articles pretty quickly but I somehow missed these two:

Networking problems the next virtualization hurdle?

Server hardware virtualization, such as that provided by VMware with VMware Infrastructure 3 (VI3), by Citrix with XenServer, or by Microsoft with Microsoft Virtual Server (and eventually Hyper-V), can make a tremendous difference in data center design.

Networking is certainly one area where the effects of virtualization are being felt.

How virtualization impacts IT staff, security and change management

There is little doubt that virtualization can offer tangible benefits, such as finally making it possible to do more with less. But adopting server virtualization technology isn’t all roses for an IT organization. In this tip, I spell out why server virtualization creates three specific challenges: breaking down IT department fiefdoms; possible increases in security vulnerabilities; and greater complexity in change management.

Feedback on the articles is welcome; I always try to be open to anything that may improve my abilities as a writer or that will help me better serve my readers.

At the start of 2007, I wrote a pair of articles on Mac FTP/SFTP clients, lamenting the performance of the open source Cyberduck application and eventually settling on Interarchy as my replacement product.

Quite a bit has happened since that time, and I now find myself in a bit of a quandary. Cyberduck has replaced the underlying file transfer engine to dramatically improve overall throughput, and Interarchy has “refined” the user interface to make it look more Mac-like. Unfortunately, in the process, Interarchy has become downright ugly. I don’t like the ugly chiclet buttons in Leopard’s Finder, I don’t like them in Safari, and I don’t like them in Interarchy, either. I very much prefer “old school” color icons in the toolbars.

Given that the new Interarchy interface was really bugging me—the Finder, at least, is still usable when you turn off the sidebar and close the toolbar—I thought I’d try Cyberduck again. Simple, one-file performance tests showed that Cyberduck, with the option to transfer files via SCP enabled, was coming much closer to matching Interarchy’s performance; Interarchy remained in the lead for raw performance, however. I figured that I could live with the performance degradation for most tasks, and started trying to use Cyberduck whenever possible.

Until tonight, that was. I needed to transfer a large number of smaller files up to one of my web sites, and Cyberduck simply dragged. I guess it’s the method whereby Cyberduck manages connections that slowed down the process, since it appeared that it checked the connection to the server for each file it was transferring. In any case, I’ve found that it looks like I’ll just have to suffer through Interarchy’s new look; the alternative is just too slow.

Some time last week, one of my NetNewsWire subscriptions turned up an article titled “HP VirtualConnect” from Brent Ozar. It turns out that Brent’s blog, while heavily SQL Server-oriented, has a few nuggets of HP Blade information:

HP C-Class Blade Chassis Review
HP C-Class Blade Chassis Review Part 2: The Cisco/Brocade Interconnect Switches
HP Virtual Connect

Of course, Brent must be pretty smart since he linked back to my site…

Seriously, though, have a look at Brent’s HP c-Class articles for a good overview of the product and its associated technologies.

Back on March 5 blogger extraordinaire Alessandro Perilli of virtualization.info revealed that Cisco had chosen KVM as the virtualization platform for IOS-XE, the new Linux-based version of IOS that runs on the recently introduced ASR series of routers.

If you were like me, you may have been wondering exactly how Cisco was putting KVM to use. No need to wonder any longer! Colleague and fellow blogger Colin McNamara has written up a detailed and in-depth discussion of the ASR1000 and how it uses KVM to provide virtualized instances of IOS-XE. Colin also discusses the role of the QuantumFlow processor and, believe it or not, the role of Popeye and Spinach. (Go read the article. It will make sense when you’re done.) Nice work, Colin!

This is a handy little trick that many of you may already know. Starting with version 3.5, VMware added support for Cisco Discovery Protocol (CDP) on the ESX Server vSwitches. CDP support is enabled on a vSwitch with this command:

esxcfg-vswitch -B both vSwitch0

The “-B” parameter is case-sensitive; the “-b” (note the lowercase B) displays CDP status while the “-B” (uppercase B) configures CDP.

Once CDP support is enabled on the vSwitch—and assuming it is enabled on the physical switch—running the “show cdp neighbor” IOS command will show the link between each physical switch port and the matching ESX Server NIC. The output will look something like this:

Capability Codes: R-Router, T-Trans Bridge, B-Source Route Bridge
                  S-Switch, H-Host, I-IGMP, r-Repeater, P-Phone

Device ID Local Intrfce Holdtme Capability Platform      Port ID
s3        Gig 0/26        147      T S     WS-C3524-XFas 0/24
esx04     Gig 0/22        168       S      VMware        ESXvmnic0
esx04     Gig 0/21        168       S      VMware        ESXvmnic1

As you can see in the output above, the CDP output clearly links the physical switch port and the ESX Server NIC. This makes it incredibly easy to identify the NICs in the server. This is particularly helpful in blade situations, since you can’t exactly unplug the NIC and see which one goes down with “esxcfg-nics -l” (a common approach to identifying the NICs in the server). Of course, this requires Cisco switches in the blade chassis. Since the internal port mappings on the blade chassis determine which NICs connect to which ports, this command adds the mapping within ESX Server and lets us quickly and definitively identify the NICs in the server as seen by ESX Server.

As a follow-up to my earlier post about using VLANs with VMotion, I wanted to also share a brief configuration snippet (based on a Cisco IOS-based switch) that aligns with the recommendations in that article. This is nothing new to those who are familiar with IOS, but for those readers who are not it may provide some helpful information.

For the purposes of this configuration, I’m making the following assumptions:

• VLANs 100 and 200 are the VMotion VLAN and some other production VLAN, respectively (perhaps a VLAN carrying virtual machine traffic, or a VLAN carrying Service Console traffic)
• VLAN 4094 is the “ESX Trunk” VLAN, which isn’t used anywhere else in the network for any purpose

Here’s the recommended port configuration for ports connecting to an ESX Server:

interface g0/1
switchport trunk encapsulation dot1q
switchport trunk native vlan 4094
switchport mode trunk
switchport trunk allowed vlan 100,200,4094
switchport noneg
spanning-tree portfast-trunk

Technically, the “switchport noneg” command won’t really do anything; it disables DTP (Dynamic Trunking Protocol) but DTP isn’t supported by ESX Server anyway.

A couple of notes about this configuration:

• Refer back to my article on the native VLAN with ESX Server; by setting the native VLAN to 4094 (the “ESX Trunk” VLAN), you won’t carry that traffic into the ESX Server. If used on a vSwitch that also carried Service Console traffic, then it could impact automated build scripts.
• If you needed to combine this configuration with EtherChannel, refer back to my original article on ESX Server, NIC teaming, and VLAN trunking.

Keep in mind the other recommendations as well: explicitly control trunking to other devices and explicitly control the transmission of VLANs across trunks to control “VLAN leakage”.

CCIEs and other experts, I’d welcome any other suggestions as well as recommended commands to use in the switch configuration to help maximize security and minimize exposure to VLAN-based attacks.

Xensploit, as it’s called, is the recently demonstrated exploit that allows virtual machines (VMs) that are “in flight” during a live migration (XenMotion in Citrix XenServer, VMotion in VMware ESX Server) to be manipulated. If you haven’t yet read the PDF that describes Xensploit, I highly encourage that you take a look at it. It’s very enlightening as to exactly what can be done to an in-flight VM.

Naturally, the best way to protect against this particular problem is to guard the integrity of the live migration network. For simplicity’s sake, I’ll refer to this as the VMotion network from this point on, but keep in mind that it is equally applicable to any network connections on any virtualization platform that uses live migration.

The most surefire way to protect the VMotion network is to place it on its own dedicated, physically separate network, using separate physical NICs plugged into separate physical switches that do not possess any connections to production networks. This will ensure that unauthorized access to the VMotion network is prevented, but comes with disadvantages as well: this configuration requires more physical NICs and more physical switches than other configurations.

In implementations with limited numbers of physical NICs, however, this isn’t really an option. In these cases, the use of Layer 2 VLANs and multiple port groups on a single ESX Server vSwitch to allow VMotion traffic to share the same physical NICs and the same physical switches as other traffic is a very common solution. In fact, it’s a solution that I’ve recommended many, many times. But does this configuration provide enough protection for the VMotion network?

The real question is, does a simple Layer 2 VLAN offer enough protection? That question, in turn, spawns other questions: what kinds of attacks are there against Layer 2 VLANs? Is it possible for traffic to hop across VLAN boundaries?

Armed with those questions, I set out to do some research. You can see some links in my del.icio.us bookmarks that pertain to the research I did. Basically, I found that Layer 2 VLAN attacks boil down to two basic types:

1. The first type involves a malicious host pretending to be a switch and forming a 802.1Q VLAN trunk with the real switch, which then passes traffic from all VLANs across to the malicious host.
2. The second type involves double-tagged 802.1Q frames and the native VLAN, whereby traffic can, under specific circumstances, hop from one VLAN to another without any Layer 3 routing involved.

(Network and security gurus out there feel free to elaborate or correct me on this information.)

The best way to address attack vector #1 is to explicitly disable automatic trunk negotiation on all ports that don’t need to be trunks. From my research and my (relatively) limited knowledge of Cisco IOS, this command should do it:

switchport mode access

This explicitly forces the switchport into a state where it will not negotiate an 802.1Q VLAN trunk with another device, hence killing attack vector #1 dead in its tracks. Again, this should only be done on the ports that are not connected to the ESX Servers; otherwise, you’re shooting yourself in the foot. Keep in mind that uplinks to other switches, ports going out to IP phones, etc., may also need to be configured as VLAN trunks. Really, the issue is about controlling the creation of unauthorized VLAN trunks in order to control VLAN leakage. One of the CCIEs at the office mentioned a “switchport noneg” command, but I’m not familiar with that one; anyone have more details?

Addressing attack vector #2 is also relatively straightforward. Since the VLAN hopping exploit takes advantage of the nature of the native VLAN (which I’ve discussed before here), setting the native VLAN on the trunk ports connecting to the ESX Servers to a VLAN that is not used anywhere else in the network will prevent VLAN hopping. For example:

switchport trunk native vlan 4094

From my NIC teaming and VLAN trunking article (one of the most popular articles on the site, by the way), you’ll see that I recommended at that time the creation of a VLAN that is used only as the native VLAN for 802.1Q trunks. At that time, I didn’t fully understand why; now, after additional research, I understand why I needed to do that and I also recognize that the suggested configuration provides a layer of protection against VLAN hopping attacks.

To summarize:

• To protect against malicious hosts forming unauthorized 802.1Q trunks, disable automatic trunk negotiation and explicitly/manually create trunks. The key here is to ensure that VLANs don’t inadvertently “leak” beyond where they should (also see note below about specifying the allowed VLANs).
• To protect against VLAN hopping, create a VLAN that is used only as the native VLAN on the 802.1Q trunks connecting to the ESX Servers. This VLAN must not be used anywhere else in the LAN. Set this VLAN as the native VLAN on the trunks into the ESX Servers.

In addition, my networking mentors also recommended the “switchport trunk allowed vlan” command to specify which VLANs are allowed to cross 802.1Q VLAN trunks. This will help ensure that VMotion traffic is limited to only those switches that absolutely must carry it; again, we’re seeking to control VLAN leakage.

With these configurations in place, using a Layer 2 VLAN to carry VMotion traffic on the same physical NICs and same physical switches as other types of traffic is fairly well-protected against malicious interference. While it is not as secure as a physical separate, dedicated network, it is secure enough for most organizations and the reduction in infrastructure needs generally outweighs the risks.

More information and discussion about Xensploit—and protecting against Xensplot—at the following links:

Keeping Your VMotion Traffic Secure
Two vulnerabilities found in VMware virtualization products
‘Live’ VMs at Risk While in Transit
News Flash: If You Don’t Follow Suggested Security Hardening Guidelines, Bad Things Can Happen…

UPDATE: I’ve updated the wording above to more properly reflect the goal behind the use of the “switchport mode access” command, and when it should be used. Colin, thanks for the feedback and clarification!

In early January, I wrote an article about how unimpressed I was with VMware’s Remote CLI, the tool released along with ESX Server 3i and—at the time—the only way to perform a Storage VMotion operation. Since that time, of course, a Storage VMotion plugin for the VI Client has been released.

About a month later, I joined with Bob Plankers of LoneSysAdmin in expressing some disappointment about VMware Infrastructure 3.5. I also vented some frustration about issues trying to import the Remote CLI appliance into VirtualCenter from an OVF file, and problems with jumbo frames and iSCSI/NFS.

Of course, this isn’t the first time I’ve expressed concerns about VMware’s products, despite claims that I am a VMware fanboy. (I do admit to being a Mac fanboy. I’m currently receiving professional help.)

The point of this story—and yes, there is a point—is what happened after those posts. Within just a few days, VMware contacted me. No, not to hound me for speaking out against them, but to thank me for “keeping them honest” (their words) and to offer assistance in working with me to resolve these issues. Since that time, I’ve spoken with several Product Managers and Sr. Product Managers within VMware, all of whom seem genuine in their desire to make sure that the issues I experienced aren’t the results of bugs in the product or, if they are bugs, to identify them and fix them.

I have to say that’s pretty impressive. It’s highly doubtful that VMware can or does respond to every customer in the same fashion, although I wish they did, but it does occur to me that at least they’re trying. I suppose I could be cynical and say that they’re only behaving this way because they don’t want to look bad, but who am I to question their motivations? I’ve said before there’s nothing wrong with a company that’s out to make money—it does make the economy run, after all—but what really matters is the means by which the company seeks that goal.

My final article in the virtual desktop infrastructure (VDI) series at SearchVMware.com has been published!

From the article, “Networks, host OSes strained by VMware VDI deployments”:

Deploying Virtual Desktop Infrastructure (VDI) on VMware Infrastructure 3 (VI3) can have a considerable impact on your network design and hosted operating system (OS) instances.

The article goes on to discuss specific ways in which the network and the guest operating systems have to be architected in a VDI deployment. I hope you find the information useful!

In Virtualization Short Take #2, I mentioned the announcement by Catbird of their HypervisorShield, and joined with Christofer Hoff in asking what it was, exactly, that Catbird was seeking to protect with this product.

Since that time, Christofer had the opportunity to speak with Michael Berman, CTO for Catbird, and get some additional information and clarification on what Catbird’s new product is and isn’t doing. This clarification is available here.