June 2007

Network Appliance storage systems support the use of virtual interfaces (VIFs) to provide link redundancy and improved network throughput.  Two types of VIFs are available:

• Single-mode VIFs act like a fault tolerant team and will fail traffic over to a standby link when the active link goes down.
• Multi-mode VIFs act like a group of links providing aggregate bandwidth as well as link redundancy.

Single-mode VIFs are great for fault tolerance, but the storage system isn’t leveraging all the links.  It’s “active-passive” arrangement in which only one of the links is passing traffic while the other link is idle.  No switch support is required for this configuration.

Multi-mode VIFs, on the other hand, allow for both greater bandwidth utilization as well as fault tolerance.  Traffic will be distributed across all the links in the VIF (typically based on IP address), and if one link fails the traffic is redistributed across the remaining links.  However, this configuration requires support on the switch.  In this article, we’re going to look at configuring a Cisco Catalyst 3560 switch to do link aggregation with a NetApp storage system running Data ONTAP 7.1.1.1.

To configure the switch, we’ll use the following commands (these are entered in global configuration mode on the switch):

s3(config)#int port-channel1
s3(config-if)#description Multi-mode VIF for netapp1
s3(config-if)#int gi0/23
s3(config-if)#channel-group 1 mode on
s3(config-if)#int gi0/24
s3(config-if)#channel-group 1 mode on

This creates the port-channel1 interface (you may need to increment that number, i.e., use port-channel2 or port-channel3, if you already have existing link aggregates configured) and adds interfaces GigabitEthernet0/23 and GigabitEthernet0/24 to the link aggregate.  If you do have to use a different link aggregate interface, be sure the number of the interface (“int port-channel4”) matches the number of the channel-group specified on the member interfaces (“channel-group 4 mode on”).  This seems obvious, but it’s worth mentioning nevertheless.

Be aware that Data ONTAP’s multi-mode VIFs are only compatible with static 802.3ad link aggregation; you can’t use PAgP (Cisco proprietary protocol).  I would assume dynamic LACP is also incompatible.  For this reason we used the “channel-group 1 mode on” statement instead of something like “channel-group 1 mode desirable”.

By default, many Cisco switches default to MAC address-based load balancing across the links, whereas NetApp defaults to IP address-based load balancing.  To see the switch’s current load balancing configuration, use this command in privileged mode:

s3#show etherchannel load-balance

To change the switch’s load balancing algorith to a mode compatible with NetApp’s, use one of the following command in global configuration mode (note that changing it affects the entire switch; you can’t change it for a single port-channel individually):

s3(config)#port-channel load-balance src-dst-ip

Once the switch is configured, then we can proceed with configuring the NetApp storage system.  The following commands will create the the multi-mode VIF (this can also be done via the FilerView GUI):

netapp1>vif create multi vif0 e6d e7d
netapp1>ifconfig vif0 172.31.254.10 netmask 255.255.255.0
netapp1>ifconfig vif0 up

This creates the VIF with interfaces e6d and e7d as members, plumbs it with an IP address, and brings it up.  Running the command “vif status vif0” now will return the following results:

default: transmit ‘IP Load balancing’, VIF Type ‘multi_mode’, fail ‘log’
vif0: 2 links, transmit ‘IP Load balancing’, VIF Type ‘multi-mode’ fail ‘default’
 
VIF Status Up Addr_set
up:
e6d: state up, since 05Oct2001 17:17:15 (05:23:05)
mediatype: auto-1000t-fd-up
flags: enabled
input packets 2000, input bytes 12800
output packets 173, output bytes 1345
up indications 1, broken indications 0
drops (if) 0, drops (link) 0
indication: up at boot
consecutive 3, transitions 1
e7d: state up, since 05Oct2001 17:18:03 (00:10:03)
mediatype: auto-1000t-fd-up
flags: enabled
input packets 134, input bytes 987
output packets 20, output bytes 156
up indications 1, broken indications 0
drops (if) 0, drops (link) 0
indication: broken

Note the ‘IP Load balancing’ algorithm stated in the output; this is why the switch’s load-balancing mechanism should be changed to match.

At this point, the links should be up between the Cisco switch and the NetApp storage system, and traffic should be passing to and from the storage system without any problems.  To test the fault tolerance, we can pull one of the links in VIF; traffic should continue to flow with very little, if any, interruption.  And while traffic from a single client to the NetApp won’t see a significant increase in throughput, the overall throughput of multiple separate clients to the NetApp should improve with multiple links in the VIF.

More information, including additional Cisco configs, is available here.

For the last few weeks, I’ve been implementing a Squid proxy server (with SquidGuard content filtering) to help control outbound web access from my home network.  Basically, I wanted to make sure that the kids weren’t accessing content that was inappropriate.  So, after making sure that the proxy server was working as expected, last night I locked down the Cisco PIX firewall (OK, I’m a geek—what can I say?) to only allow outbound HTTP/HTTPS traffic from the proxy server itself.  (I suspect that one of my daughters had discovered that she could bypass the proxy, hence the need to lock down the firewall.  She’s got a surprise coming.)

Upon arriving home from work today, I booted up my laptop and launched my suite of applications and sites (Camino, NetNewsWire, Adium, Cocalicious, webmail for the office, a few other key web sites, etc.).  I was greeted with a prompt to enter the password for two MSN accounts that I use for instant messaging with Adium.  I entered the password.  It prompted me again.  Puzzled, I typed the password again, more slowly to make sure that I had every character right.  Still wouldn’t connect.  What was going on?

Now suspicious that someone had defrauded my accounts, I logged in to the accounts via an HTTPS connection in a Web browser.  No, the password was right; the accounts were OK.  So why wouldn’t Adium connect?

Next, I reviewed the account settings in Adium while also performing a Google search.  The settings looked right (most notably, the “Connect via HTTP” option was unchecked), and the Google search turned up an Adium Trac ticket about MSN connectivity with a proxy server.  At that point, the light bulb came on—I must need to configure Adium to use the systemwide proxy settings in Mac OS X.  A quick couple of clicks later I was done, but Adium still wouldn’t connect.  Huh?

<aside>I still haven’t determined if configuring Adium to use the systemwide proxy settings failed due to a content violation, an error with the proxy configuration, or a bug in Adium.  Nothing was logged in the blocked sites log on the proxy, though, so I’m leaning toward one of the latter two.</aside>

OK, then, let’s go old-school on this thing.  I logged into my PIX firewall, issued a “show access-list” command to get the current traffic matching statistics, and tried to connect again.  Another “show access-list” command, and it became painfully clear that the outbound rule blocking HTTPS access from everyone except the proxy was causing the problem.

<rant>If an application says that it uses a certain set of protocols or ports, DON’T use an entirely different set of protocols and ports.  It drives people like me crazy!  (Not to mention it flies in the face of trying to establish reasonable security policies.)</rant>

Back in the PIX firewall again, I entered a “debug packet” command to echo HTTPS packets exiting the network, and tried connecting to my MSN account from Adium again.  Yep, there it is—a couple of different IP addresses (65.54.179.228 and 65.54.183.203) showed up in the debug output every time I tried to connect.  After modifying the access-list to allow HTTPS traffic to those specific addresses, Adium connected without any further problems.

Now what would an “ordinary” person have done in this situation?  Would an “ordinary” (and by that I mean non-network engineer or non-geek individuals) person have known to check their firewall?  Or to issue a command to have the firewall tell you about blocked packets so that they could determine exactly what was being blocked and why?

Granted, ordinary people probably won’t have PIX firewalls at their house, and probably wouldn’t be locking down outbound traffic via access lists.  Still, doesn’t this tell us something?  Shouldn’t applications adhere to the protocols and ports they say they are using?  Shouldn’t applications be intelligent enough to provide an error message that describes the underlying problem?  After all this progress in computer hardware, applications, and networking, and we are still unable to accurately diagnose problems without going to multiple sources?  Or am I just expecting too much?

Just yesterday, I was asking if Unity—the Coherence-killing feature that was soon to be added to VMware Fusion—was real or rumor.  (Of course, yesterday I was also getting updated by the Fusion developers that this is indeed not rumor and is definitely real.)  Now I’m here to tell you that I have experienced it with my own eyes.

VMware Fusion Beta 4 has been released (get it here), and with it comes Unity.  In a word: Wow!  It’s pretty disconcerting to see Windows applications (like a command prompt, Internet Explorer, or Visio) running side-by-side with Camino, Mail.app, and NetNewsWire.  Interleave windows?  Not a problem.  Drop shadows?  There.  Minimization effects?  Yep.  Aside from the window decorations, you would be hard-pressed to tell which applications were running natively in Mac OS X or in virtualization.

And that, my friends, is the heart of what I described ages ago (OK, perhaps it only seems like ages ago) as application agnosticism.  With virtualization technology and UI integration like this, who cares what OS an application is written for?  Of course, we still have a ways to go to fully realize application agnosticism, since Unity is only available for Mac OS X (not Linux or Windows), but this is a powerful step in the right direction.

My hat is off to the Fusion developers.  Well done!  A lot of people were predicting that Parallels had too much of a lead in Mac virtualization, but all of you have definitely proved them wrong.  Keep up the good work!

<aside>Boring technical question to any VMware developers, Fusion team members, etc:  Is Unity debuting on OS X because of some advantage within Mac OS X itself, or simply because of the timing and the competition?  I am a Mac fan, after all, and was curious if my favorite OS had a leg up on the competitors that made it easier to create Unity.</aside>

There’s a lot of buzz about Unity, a future VMware Fusion feature that, if real, completely trumps Parallels’ Coherence functionality.  The question: Is Unity real or fake?

The VMTN Blog posting “Coherence? Meet Unity” seems to be the source of it all, which lends a certain air of credibility.  The idea was quickly picked up by The Unofficial Apple Weblog (TUAW), CrunchGear, and YouTube, among others.

However, this is an apparent about-face from the position of Fusion developers earlier in the product’s lifecycle, who were not so keen on implementing such a feature in Fusion until “they could do it right.”  Has that time finally come?

I will say this:  if Unity is truly real and coming in a future release of Fusion, and if the features that were demonstrated in the Unity video are real, it completely blows away Coherence.

UPDATE:  According to Regis (VMware Fusion lead developer), this is definitely real.  I stand corrected!

Two of my favorite applications have been updated recently.  NetNewsWire has reached version 3.0 (and adds lots of new features), and Camino reaches version 1.5.

The NetNewsWire update brings, among other things, native Growl support (no need for NewsGrowl any longer), new display and layout options, a slightly more streamlined interface, and Spotlight support.  I just installed the new version this morning, and I’ve already found the Spotlight integration to be useful.

The Camino update brings an updated rendering engine (Camino 1.5 now shares the same version of Gecko as Firefox 2.0), enhanced ad/pop-up blocking, and RSS feed detection.  Nothing earth-shattering, but useful features nonetheless.

If you use either of these applications, I suggest you upgrade to the latest versions.  I don’t think you’ll be disappointed.

Think about it:  Does it really matter to Microsoft if a server running Windows is physical or virtual?  If a company buys ten servers running Windows Server, then Microsoft gets revenue for ten licenses of Windows Server.  If a company buys two servers running VMware ESX Server but has ten guests running Windows Server, Microsoft still gets license revenue.  So what does it matter to Microsoft, as pointed out in this article:

Customers will be using VMware or Virtual Iron, for example, and they will buy Windows and other Microsoft software to run on those virtual machines.

A VMware Virtual Infrastructure deployment does not preclude the use of Microsoft software; in fact, it often accelerates the deployment of more instances of Windows.  How many times have you read about “virtual server sprawl”?  I would wager that the majority of those sprawling virtual servers are still running Microsoft Windows.  So why is Microsoft fighting so hard in the virtualization space?

Why try to create your own hypervisor to compete with VMware’s wildly successful hypervisor in ESX Server?  Why not partner with VMware to make your products even better when running on VMware’s virtualization layer?  Why not build a management console that can manage VMware’s virtualization technologies better than their own management console?  I mean, if the real challenge is the management of VMs, why not tackle that?

Why not “embrace and extend”?

There’s a small gotcha when using Microsoft’s iSCSI initiator and MPIO driver to do iSCSI multipathing:  the Microsoft initiator and MPIO driver will overwrite the IQN of the iSCSI HBA.  Obviously, this could cause problems where access control to iSCSI LUNs is based on initiator IQN.

As pointed out in this Qlogic support document (check the “Additional Notes” section at the bottom of the page), installation of the Microsoft iSCSI initiator will overwrite the IQN of the HBA with a Microsoft-generated IQN, like “1991-05.com.microsoft:servername.domain.com” or similar.

In environments where access to LUNs is controlled in part or in whole by initiator IQN, this is a problem.  One such environment is NetApp iSCSI SANs, where initiator groups (or “igroups”) control access to LUNs based on the IQNs of the initiators.  To work around this, you’ll want to add the original IQNs of the HBAs (before the installation of the Microsoft iSCSI initiator) as well as the Microsoft IQN in the igroups for the LUNs that should be visible to that server.  Otherwise, you could lose access to the LUN after installation of the Microsoft initiator.

(By the way, in case you’re wondering why one would install the Microsoft iSCSI initiator when you’ve already got HBAs, there’s a good reason—to get multipath support.)

To be honest, there are times when using Mac OS X can be difficult.  With the overwhelming installed base of Windows and the attention of the media darling Linux, vendors will often provide instructions on how to do something with Windows or Linux, but not from Mac OS X.  Sometimes the Linux instructions work, but many times they don’t, and the Windows instructions typically involve running some sort of Windows executable that won’t, of course, run on Mac OS X.  What is one to do, then?

This is situation in which I found myself last week while trying to upgrade a Network Appliance storage system (an old F840) in the lab.  I wanted to upgrade the F840 to Data ONTAP 7.1.2.1, the latest GA release in the 7.1.x family (Data ONTAP 7.2.x isn’t supported on the F8xx series).  There are instructions for performing the upgrade from a UNIX/Linux-based system as well as instructions for upgrading the storage system from a Windows-based computer.  OK, but what about a Mac?

Well, Mac OS X is based on FreeBSD, so the UNIX instructions should work, right?  Well, not quite.  Although I mounted the NetApp’s root volume via NFS and attempted to extract the files per the instructions, it kept failing.  OK, what if I mount it via CIFS?  Same result—an error during the file extraction/file copy process and an error when running the “download” command.

Well, I can’t exactly run the Windows executable version of the Data ONTAP upgrade because I don’t run Windows.  The UNIX/Linux instructions don’t work.  Fortunately, there’s a third option—the HTTP server upgrade.  I had an OpenBSD-based web server in the lab, so within just a few minutes I had the Data ONTAP upgrade files on that server and then onto the storage system itself using the “software get” command.  The process worked quite smoothly, and I was finished upgrading the storage system a few minutes later.

I even went back and reviewed the upgrade guide again to see if it explicitly listed specific client operating systems as a prerequisite for upgrading the storage system.  The upgrade guide states that “any available version of UNIX” is required.  So why didn’t Mac OS X work?  Why did I keep getting errors running the Perl install script, yet the HTTP upgrade worked flawlessly?  Don’t get me wrong, I’m glad the upgrade worked.  I just a bit peeved that something that should have worked according to the documentation didn’t.

So, here’s a tip for all of you out there: if you ever need to upgrade a Network Appliance storage system to a newer version of Data ONTAP from a Mac OS X system, be sure to use the HTTP upgrade method.

This IBM BladeCenter H installation I’ve been working on with another engineer for the last couple of days is not going as smoothly as we both would have liked.  I don’t know if this is indicative of the BladeCenter H chassis itself, or if it’s just me.  While some would say it’s just me, I suspect it’s a little of both.

For example, we ran into issues with the Management Modules “freaking out,” and failing over between the modules didn’t solve the problem.  In fact, we had to power down both management modules and then power them back up one at a time in order to get the fans to finally settle down.  Otherwise, the chassis sounded like a jet turbine getting ready to take off, and—get this—people outside the datacenter could hear the chassis.

Even after getting that settled down, there was still some weirdness with the IBM KVM switch that we still haven’t resolved (kept freaking out the keyboard and causing it to stop working).  We had to plug in a separate USB keyboard in order to work with the chassis.  That particular issue still hasn’t been resolved.

The real kicker, though, was the problem we ran into with the floppy drive.  The design called for boot from SAN via iSCSI (using LUNs off a Network Appliance storage system) using Qlogic HBAs.  This is a supported configuration, but requires the use of a driver floppy during the installation of Windows Server 2003.  The older BladeCenter chassis had a built-in floppy, but the BladeCenter H does not, and the USB floppy that we tried to use wouldn’t work.  No matter how hard we tried, the blade(s) just wouldn’t see the floppy drive.  Until we can get a floppy drive recognized during the Windows setup process, we can’t get Windows installed and we are just stuck.

Anyone else run into similar issues with a BladeCenter H?