blog.scottlowe.org

A friend and colleague of mine recently launched a blog of his own, The Blade Blog.  If the useful information that’s been posted there so far is any indication, this will be a definite addition to your RSS subscriptions.  Keep up the good work, Aaron!

UPDATE:  After some great feedback on the launch of his site, Aaron’s moved away from Blogger to a hosted WordPress solution and has gotten a new domain name.  You can now find Aaron at the Blade Vault.  More information and updated URLs available here and here.

It’s common in blade deployments to use multiple Ethernet switches in the blade chassis to provide network redundancy (I’ll refer to these as “chassis switches” moving forward).  For example, in both the IBM BladeCenter H and the HP BladeSystem c-Class, we can provision multiple chassis switches so that half of the NICs on the blades connect to one chassis switch and the other half connect to the other switch.  Within the OS, we load NIC teaming software to provide automatic failover if one of the links goes down.  In this scenario, if one of the chassis switches fails then traffic will automatically fail over to the other switch.

In cases like this, everything works as advertised.  But what about when the chassis switch stays up, but the uplink from that switch to the outside world goes down (perhaps the upstream switch went down or the link was unplugged)?  In that case, the link from the chassis switch to the blade’s NIC is still up, and therefore the NIC teaming software in the OS does not know that a problem has occurred and will not move the traffic to the other link.  In situations like this, we need to implement link state tracking.

<aside>Astute readers will recognize that link state tracking is actually applicable in any server deployment—not just a blade server deployment—where the servers connect to a distribution switch and not the core.  I’m just going to focus on blade server deployments here, but the configuration would be much the same, if not exactly the same, in non-blade server deployments.</aside>

Link state tracking is pretty easy to configure; you define one or more upstream ports and one or more downstream ports.  The upstream port(s) are the ports that uplink to the rest of the network; in a blade server deployment, this would be the ports (or port groups) that connect to the network backbone.  The downstream port(s) are the ports that connect back to the servers.

Here’s an example.  We have a Cisco chassis switch that has a GigabitEtherChannel port group defined as an uplink out to the outside world:

interface Port-Channel1
description Uplink to network backbone
switchport trunk encapsulation dot1q
switchport trunk native vlan 2
switchport trunk allowed vlan 2-4094
switchport mode trunk
link state group 1 upstream

Note the “link state group 1 upstream” command, which marks this port channel as an upstream port.  If all the links in this port channel go down (thus making the port channel itself go down), then the switch will notify downstream ports in the same group to mark themselves as down also.

The member ports of this port channel would not have the “link state” command present:

interface GigabitEthernet0/18
description Port group member for uplink to network
switchport trunk encapsulation dot1q
switchport trunk native vlan 2
switchport trunk allowed vlan 2-4094
switchport mode trunk
channel-group 1 mode on

So for the ports on the same chassis switch that are connecting to the servers in the chassis, we have this configuration:

interface GigabitEthernet0/10
description Web server NIC
switchport access vlan 2
switchport mode access
link state group 1 downstream
spanning-tree portfast

Note the “link state group 1 downstream” command, which marks this port as a downstream port from the Port-Channel1 interface.  If Port-Channel1 goes down (because all the member links in Port-Channel1 also went down), then GigabitEthernet0/10 will also go down.  Because GigabitEthernet0/10 went down, the NIC teaming software running in the OS on the blade will fail the traffic over to a different NIC, presumably a NIC that connects to the redundant chassis switch.

You’ll also need the global “link state track 1” global command to enable link state tracking (thanks for the clarification, Matt!).

Because of the nature of blade deployments, this sort of configuration is particularly applicable in blade deployments, but also applies in other situations as well (as mentioned earlier).  I hope this is useful!

This IBM BladeCenter H installation I’ve been working on with another engineer for the last couple of days is not going as smoothly as we both would have liked.  I don’t know if this is indicative of the BladeCenter H chassis itself, or if it’s just me.  While some would say it’s just me, I suspect it’s a little of both.

For example, we ran into issues with the Management Modules “freaking out,” and failing over between the modules didn’t solve the problem.  In fact, we had to power down both management modules and then power them back up one at a time in order to get the fans to finally settle down.  Otherwise, the chassis sounded like a jet turbine getting ready to take off, and—get this—people outside the datacenter could hear the chassis.

Even after getting that settled down, there was still some weirdness with the IBM KVM switch that we still haven’t resolved (kept freaking out the keyboard and causing it to stop working).  We had to plug in a separate USB keyboard in order to work with the chassis.  That particular issue still hasn’t been resolved.

The real kicker, though, was the problem we ran into with the floppy drive.  The design called for boot from SAN via iSCSI (using LUNs off a Network Appliance storage system) using Qlogic HBAs.  This is a supported configuration, but requires the use of a driver floppy during the installation of Windows Server 2003.  The older BladeCenter chassis had a built-in floppy, but the BladeCenter H does not, and the USB floppy that we tried to use wouldn’t work.  No matter how hard we tried, the blade(s) just wouldn’t see the floppy drive.  Until we can get a floppy drive recognized during the Windows setup process, we can’t get Windows installed and we are just stuck.

Anyone else run into similar issues with a BladeCenter H?

In a recent article discussing Novell’s leadership change, one analyst was quoted regarding the change as being positive for Novell in that they (Novell) could stop building very high quality products and instead build products that are just good enough.  I don’t know about you, but this spirit of mediocrity is exactly the wrong kind of thinking for IT vendors.

Specifically, the quote stated this:

“Ron Hovsepian appears to be an astute business person, one who will be able to quickly take stock of the environment and Novell’s position within that environment. This, I hope, will help Novell move from its current position of very slowly building extremely high quality products to quickly building and marketing products that are good enough to satisfy the market,” concluded Kusnetzky.

So what is he (Dan Kusnetzky) proposing then?  It sounds to me that Dan thinks IT software vendors should make their products just good enough to pass muster, instead of making them the best that they can be.

In my opinion, this spirit of mediocrity—this willingness to accept products that are knowingly released with imperfections and flaws because they are “good enough”—is exactly what brought the industry to where it is today.  This mediocrity is what brought SQL Slammer, Blaster, and Melissa.  This is the view that accepts that rebooting your computer a few times a day is just a part of life, and that our operating systems and applications shouldn’t be expected to be stable and reliable.  Just good enough?  When was the last time you recommended a product, service, or vendor because they were “just good enough”?  No, just good enough isn’t good enough.

Every major IT vendor out there—from HP, IBM, and Sun, to Apple, Microsoft, and Red Hat—should be held accountable for the quality of the products they release.  Hey, I understand that companies may make mistakes, and miss errors.  That’s understandable.  But any company that knowingly releases a product that’s “just good enough” when it could have been better is not a company we should be praising.  We should be supporting those companies that emphasize quality over “just good enough”.

Perhaps I’m overreacting.  Perhaps the analyst’s comments were merely directed at the speed with which Novell releases their products, and was instead trying to state that Novell needed to release competing products more quickly.  Even so, any vendor that values speed over quality is bound to get bitten sooner or later.  Microsoft got bitten, and changed their priorities (somewhat).  Apple will get bitten, too, if they start letting the quality of Mac OS X releases slide in favor of shorter development cycles.  The same goes for all the other vendors.

What about you?  I’d love to hear your comments on the matter.

There are lots of industry pundits out there proclaiming that the introduction of Boot Camp—Apple’s new beta application that simplifies and streamlines the installation of Windows XP (and presumably Windows Vista as well) on Intel-based Macs—is simply the first step in a complex scheme that will eventually culminate in something much bigger.  I’m not so sure about that.

The predictions range the whole gamut of possibilities.  Some are predicting that Apple will begin reselling Windows XP pre-loaded on its Intel-based Macs, much in the way that MacMall is now doing.  In this scenario, Apple differentiates itself from other x86 vendors in that it offers the only solution that will also run Mac OS X.  This is something that Dell won’t be able to do.

Others are predicting that Apple will implement a Win32-compatible API, such as Darwine, so that Mac OS X will become a “better Windows than Windows” (quotes mine).  Does anyone remember OS/2?  Towards the end of OS/2’s life, IBM started positioning OS/2 as a better way to run Windows applications than Windows.  Not too terribly long after that, OS/2 died.  Will Mac OS X follow the same fate?  No, there are too many differences between OS/2 and Mac OS X (and between IBM and Apple) to believe that these two technologically superior operating systems will take the same path.  However, I do believe that it would be a mistake for Apple to try to position Mac OS X as a better way to run Windows applications than Windows itself.  Microsoft’s “embrace and extend” philosophy rarely works against them, and often backfires.

The most likely approach involves virtualization.  Making it possible to run an instance of Windows under Mac OS X (using a hypervisor and built-in Intel VT technology), so that users can run those “legacy” Windows applications that don’t have a native Mac equivalent.  This makes the switch seamless and no risk.  (Note further that Boot Camp additionally reduces the risk of switching, since a user can go back to Windows whenever needed).

I could be way off here; it certainly wouldn’t be the first time.  What do you think?

There is an incredible amount of momentum growing around the open source Xen hypervisor, and it is increasingly looking like market leaders VMware and Microsoft should be less worried about each other and more worried about Xen.

Check out some of the recent news articles regarding Xen:

“Novell to integrate Xen 3.0 in the next Open Enterprise Server”
<http://searchopensource.techtarget.com/columnItem/0,294698,sid39_gci1176440,00.html>

“Virtual Iron annonces 3.0 commercial and free editions based on Xen”
<http://www.virtualiron.com/news_events/releaseDate-4-03-06.cfm>

“Red Hat Formally Announces ‘Integrated Virtualization’”
<http://www.redhat.com/about/news/prarchive/2006/virtualization.html>

“Virtual Iron, XenSource to Unveil Xen 3.0 Products at LinuxWorld”
<http://www.eweek.com/article2/0,1759,1945398,00.asp>

(Note:  all links are courtesy of virtualization.info.)

And those links are just from the last few days!  Clearly, there is lots of momentum and lots of support from big name vendors such as HP, IBM, Novell, Red Hat, and others around the Xen open source hypervisor.  While some have speculated that VMware’s move to release VMware Server for free (and Microsoft’s corresponding drop in price for Virtual Server 2005 R2) have been to stave off each other, perhaps their moves were in response to Xen instead?

NewsForge ran this article about FOSS (free and open source software) for OS/2.  It was a real eye-opener; I hadn’t even considered the possibility that the kind of grassroots support that has driven the support and development of operating systems such as Linux and the BSDs (OpenBSD, FreeBSD, and NetBSD) might also work for OS/2.

I used to run OS/2, years ago, on one of my very earliest computers.  I said then, and I’ll say again today, that OS/2 was light-years ahead of anything else available at the time.  Sadly, it’s not always the best technology that wins, and let’s just say that Microsoft has always been good at marketing.  (I think both IBM and Novell have seen the strength of Microsoft’s marketing muscle.)  The object-oriented WorkPlace Shell was everything that Windows had always wanted to be but never quite made it.  And virtual DOS machines—the ability to boot multiple, distinct DOS versions in separate, isolated environments hinted at the virtualization trend that is now taking everything by storm.  Of course, this was years and years before VMware ever came into existence (but not before IBM was doing virtualization on the mainframe; see this article for examples).

Alas, OS/2 never got the treatment it rightfully deserved.  Languishing from a dearth of native applications, plagued by hardware compatibility issues, bungled by horrible marketing and support from IBM, and going up against the Microsoft juggernaut that was Windows 95, it hardly stood a chance.  But it’s nice to see how die-hard OS/2 fans are continuing to support the operating system even now.