April 2006

One topic that I had been very interested in exploring was the idea of VLAN tagging within ESX Server.  ESX Server offers the ability to establish 802.1q trunks with compatible switches so that 802.1q tagged frames can be routed to appropriately configured “port groups” within ESX Server.  This would allow a single physical host server to run virtual servers in multiple VLANs without requiring a separate physical connection for each VLAN.

The idea of having an ESX Server create an 802.1q trunk (even using multiple physical connections for redundancy) becomes really important when you move into the larger-scale server consolidation projects.  Organizations just won’t have the switch density (or the NIC density on the ESX Server, for that matter) to have multiple physical connections to each VLAN.  The ability to “extend” these VLANs into ESX Server’s virtual networking capabilities is very important.

Today, I set out to test this interoperability.  A host server running ESX Server 2.5.3 (the latest version, just released a few days ago) was connected to a Cisco Catalyst 3524XL running IOS 12.0(5).  The following commands were used to configure the port to which the ESX Server was connected:

Switch(config-if): switchport mode trunk
Switch(config-if): switchport trunk encapsulation dot1q

A quick review of the output from “sh int fa0/18 switchport” (where “fa0/18”, for example, is the interface to which the ESX Server is connected) showed that the port was indeed configured and operating as an 802.1q trunk.

Next, a new VLAN (VLAN ID 10) was created on the switch.  No ports were placed into this VLAN because no physical ports were required; the ports would all be coming from ESX Server across the trunk.

Third, port groups were configured in the MUI for ESX Server.  Two port groups were configured, one to match the native VLAN on the switch (VLAN 1, typically) and one to match VLAN 10.

Once all of the configuration was done, testing began.  Upon moving the first VM to the default port group (this was the port group created to match the native VLAN), I lost connectivity to that VM from other systems connected to other ports also in the native VLAN.  Thinking that perhaps all VMs needed to configured for a port group (some VMs were still connected to virtual switches and not port groups), I added the second running VM to the default port group as well.  Connectivity was established between the two VMs within the same port group, but I still had no connectivity to other systems on the same VLAN.

After a fair amount of troubleshooting and searching, I came across a reference on the VMware web site regarding the use of native VLANs.  VMware recommends against the use of native VLANs because switches tend to strip off the 802.1q tags for frames on the native VLAN.  To make VLAN tagging in ESX Server work in this situation, the switch must be reconfigured to tag all frames, including frames in the native VLAN.

The command in Cisco IOS to do this is “vlan dot1q tag native”.  Unfortunately, this command is not supported on the version of IOS that is running on the Catalyst 3524XL I used in the test lab, so there was no way to make VLAN tagging work when using the native VLAN.  This is a key “gotcha”—make sure that the switches support tagging native VLAN traffic if you plan to use native VLANs with ESX Server and port groups, otherwise it isn’t going to work.

As soon as I have the opportunity to upgrade the IOS image on the Catalyst 3524XL switch in the test lab, I’ll try testing port group/VLAN interoperability again and post results here.

UPDATE:  I’ve posted some updated information and more comprehensive configuration notes in a posting titled “ESX Server, NIC Teaming, and VLAN Trunking”.

As part of ongoing interoperability testing with ESX Server, I tested running FreeNAS (version 0.65) on ESX Server 2.5.3 today.  Since FreeNAS is based on FreeBSD (which VMware states is a supported guest operating system for ESX Server), I didn’t really expect any major surprises.  I was wrong.

Basically, I couldn’t make it work.  Despite trying both the BusLogic SCSI adapter (the default) and the LSI Logic adapter, the FreeNAS distribution wouldn’t see the virtualized SCSI hard disks, and therefore I could never install FreeNAS to the hard disk for use.  Without the XML configuration file on a read/write media, FreeNAS lost its settings (such as the IP address or the network interface to use) every time it rebooted.

It’s odd that FreeBSD, a supported guest OS, didn’t appear to work, yet I was able to make OpenBSD (an unsupported guest OS) run without any major problems.

Anyone have any tips for making FreeNAS 0.65 work under VMware ESX Server?

VMware has announced that it is making its virtual machine disk (VMDK) format openly available, downloadable and free of charge.  This introduction sets the stage for a battle for open virtual disk formats, with the major players being VMware and Microsoft.

VMware, being the market leader, seeks to continue its leadership position by fostering the creation of a thriving third-party market for add-ons to its core products.  By opening its VMDK format, VMware allows other companies to create add-ons to its virtualization products.  As the number of third-party companies creating add-ons grows, the customer base grows and attracts more developers, etc.  This self-sustaining community then helps drive adoption of VMware’s products.

On the flip side, however, software behemoth Microsoft is licensing its virtual hard disk (VHD) format to other companies.  Surprisingly enough, one of the companies to adopt the VHD format was XenSource, the commercial company behind the development of the open source Xen hypervisor.  This creates a bit of an odd alliance—we have a group of companies that generally align themselves against Microsoft (HP, IBM, Novell, and Red Hat, among others) adopting a virtual hard disk format, VHD, that is owned by Microsoft.  It would have made more sense for these companies to adopt VMware’s VMDK format (especially considering that the licensing conditions from VMware seem much more open source friendly) in order to help prevent Microsoft from taking over yet another market by bundling software into Windows.

Finally!  With the introduction of Parallels Workstation for Mac OS X (official announcement here), Intel-based Mac owners have the ability to run unmodified instances of Linux and Windows on their Mac hardware.  This has been my “killer wish” ever since the introduction of the Intel-based Macs.

In addition, rumors are swirling about the introduction of a Mac OS X-based virtualization product from market leader VMware.  (Apparently it’s already running in labs at VMware.)

Now I really wish I had a MacBook Pro…

When I upgraded to Mac OS X 10.4 “Tiger” a few weeks ago, I gave up my existing RSS reader (PulpFiction) in search of a new application (this was at the same time I had to rebuild my list of RSS subscriptions).  Having finally settled on NetNewsWire, I’m now learning to love the application that at first irritated me.

When I first tried NetNewsWire shortly after switching to the Mac, I just didn’t like it.  At the time, the thing that irritated me most was the fact that headlines disappeared after a while.  (I now know that’s “normal.”)  When I discovered PulpFiction and its persistence model, it seemed like a perfect fit.  The built-in Growl integration, support for AppleScript, and filtering functionality were natural extensions.  After a while, though, I found myself dreading to check headlines, and struggling to maintain what I thought were “reasonable” lists of headlines to be retained for future reference.  Having to process every single headline—delete it, file it, e-mail it, write about it, etc.—was just too time-consuming.  Heck, I couldn’t even keep up with my e-mail inbox, much less my RSS inbox.

When I decided to switch RSS aggregators during the Tiger upgrade, the idea of persistence was no longer a plus for me.  NetNewsWire’s model (and the model shared by many other RSS aggregators) was now appealing to me, instead of irritating.  Even so, it wasn’t love at first use; I was initially bothered by the lack of filters and built-in Growl integration.

As I soon discovered, though, the Growl integration was easily solved using NewsGrowl, and that in itself underscored a feature of NetNewsWire that I have yet to fully explore—script subscriptions.  And filters were proven to be unnecessary due to the lack of persistence in the headlines.  While I didn’t get Spotlight support (i.e., the ability to search headlines using Spotlight), I did get AppleScript support and Automator support.

In addition, it’s the little features that have really started to grow on me (note that some of these features are only in the latest beta version).  Notably:

• The right-click menu includes commands not only to copy the URL, but to also copy the headline itself—very handy for my typical workflow.
• NetNewsWire has a “Post to del.icio.us” command that works with Cocoalicious, instead of forcing me to post via a browser.  (Of course, it also integrates with ecto, my blogging client, for posting to this weblog.)
• Selective persistence, via flagging articles, allows me to choose which headlines (if any), I’d like to hang onto for a bit longer.

I’m confident that as I continue to use NetNewsWire, I’ll find more “nuggets of usefulness” built into the application.  If you’re looking for a good Mac OS X RSS aggregator, you would be well-served to give NetNewsWire a serious look.

One more quick lesson learned from the recent experiment getting OpenBSD running on VMware ESX Server involved modifying the default operation of OpenBSD’s DHCP client, dhclient.

In this case, OpenBSD only needed to obtain the IP address and subnet mask from the DHCP server.  Specifically, OpenBSD should not obtain the default gateway, as another NIC on a different subnet would be directing traffic to a separate Internet connection.  I found that by editing the /etc/dhclient.conf file, it is possible to control the behavior of the DHCP client so that it only “listens” to certain configuration parameters passed down by the DHCP server.

For example, to have the DHCP client only pick up IP address and subnet mask, change the “request” line in dhclient.conf to look something like this:

request subnet-mask, broadcast address;

The standard dhclient.conf “request” line looks something like this:

request subnet-mask, broadcast-address, time-offset, routers, domain-name, domain-name-servers, host-name, lpr-servers, ntp-servers;

Obviously, this list can be trimmed to pick up only the items that are needed by the server.  Another neat trick is using the “prepend” statement; this allows the local client to use a value configured locally, then use the values passed down by the DHCP server.  Check the man page for dhclient.conf for more detailed information.

Once OpenBSD was running on ESX Server, there was a configuration issue that had to be addressed involving static routes.  This is probably one of those times where the OpenBSD experts are saying, “This is so simple!”, but I had to search a bit to find the answer.  So, for future reference and for the reference of those of us who are not OpenBSD experts (yet), here’s the solution.

There are actually two solutions.  The first solution involves the use of the hostname.if file.  OpenBSD maintains a separate hostname.if file for each interface, where the “.if” is replaced by the devicename.  In a virtualized instance, this would typically be “le1” for the first NIC, “le2” for the second NIC, etc.  Therefore, the first NIC would be configured using hostname.le1, and the second NIC would be configured using hostname.le2.

To add a static route to an interface, append a route command to the appropriate hostname.if file.  Here’s an example:

inet 192.168.254.254 255.255.255.0 NONE
!route add default 192.168.254.1

The key, of course, is the “!route add …” statement.  This is the piece that I needed.

You can also add similar statements to /etc/rc.local, which will do the same thing.  Note, however, that these statements cannot go into /etc/rc.conf.local.

I hope Microsoft Vista is going to address the malware problem that is plaguing Windows users worldwide right now.  Even Microsoft knows it’s bad.  How bad?  Read on.

In this article from eWeek, Mike Danseglio, a program manager in the Security Solutions group at Microsoft, is quoted as saying:

“When you are dealing with rootkits and some advanced spyware programs, the only solution is to rebuild from scratch. In some cases, there really is no way to recover without nuking the systems from orbit…”

Doesn’t this say something?  If a program manager at Microsoft says what everybody else already knows, then even Microsoft has gotten to the point where they’re admitting that Windows has a problem.

This related article, published in early December 2005, notes that as much as 20% of all malware removed from Windows XP SP2 systems are considered stealth rootkits.  Considering that some of the Internet Explorer security flaws have allowed malware to be installed by simply visiting a web site, that’s pretty serious.

Microsoft has taken an excellent first step in Vista by making sure that the browser runs in a reduced-privileges environment.  Let’s hope they don’t stop there.

To help with the increasing number of articles dealing with virtualization, a “Virtualization” category has been added, and some articles have been re-categorized into this new category.

RSS subscribers may see some “updated” articles as a result of the re-categorization.

There is an incredible amount of momentum growing around the open source Xen hypervisor, and it is increasingly looking like market leaders VMware and Microsoft should be less worried about each other and more worried about Xen.

Check out some of the recent news articles regarding Xen:

“Novell to integrate Xen 3.0 in the next Open Enterprise Server”
<http://searchopensource.techtarget.com/columnItem/0,294698,sid39_gci1176440,00.html>

“Virtual Iron annonces 3.0 commercial and free editions based on Xen”
<http://www.virtualiron.com/news_events/releaseDate-4-03-06.cfm>

“Red Hat Formally Announces ‘Integrated Virtualization’”
<http://www.redhat.com/about/news/prarchive/2006/virtualization.html>

“Virtual Iron, XenSource to Unveil Xen 3.0 Products at LinuxWorld”
<http://www.eweek.com/article2/0,1759,1945398,00.asp>

(Note:  all links are courtesy of virtualization.info.)

And those links are just from the last few days!  Clearly, there is lots of momentum and lots of support from big name vendors such as HP, IBM, Novell, Red Hat, and others around the Xen open source hypervisor.  While some have speculated that VMware’s move to release VMware Server for free (and Microsoft’s corresponding drop in price for Virtual Server 2005 R2) have been to stave off each other, perhaps their moves were in response to Xen instead?