SFTP

Quite a few organizations implementing VMware Virtual Infrastructure 3 (VI3) are not organizations with a great deal of non-Windows experience.  Given that the Service Console is based upon Red Hat Enterprise Linux, this is often a completely new environment for them that introduces unfamiliar processes and applications.

Take the process of copying files, for example.  To an administrator within an organization that is primarily Windows-based servers, the idea of not being able to “map a drive” or just use a Universal Naming Convention (UNC) path is unbelievable.  For someone like myself, who has lived in a non-Windows world for a number of years now, the idea of using SCP/SFTP to copy files to a server seems pretty normal.  Nevertheless, these new VMware administrators must now get accustomed to doing things a bit differently than they have in the past, and the use of an SCP/SFTP client is one of those things.

One big problem with the use of SCP/SFTP clients is granting SCP/SFTP access to a user typically also grants shell access.  This is a problem not only for new users but also for experienced users.  New users may get into the shell and change things that shouldn’t be changed, simply because they don’t know; experienced users may get into the shell and change things that shouldn’t be changed, because they think they do know.  Either way, shell access is something to be avoided where possible, in my opinion.

The solution?  Use an “alternative” shell called scponly:

scponly is an alternative ’shell’ (of sorts) for system administrators who would like to provide access to remote users to both read and write local files without providing any remote execution priviledges. Functionally, it is best described as a wrapper to the tried and true ssh suite of applications.

Here’s how I got scponly working on a test server running ESX Server 3.0.1 in the lab.  Please keep in mind that this method is most likely not supported by VMware, so proceed accordingly.

1. Download the scponly and rsync RPM packages from the DAG repository.  (The links point to the package page, not the actual package itself, so these are not blind download links.)  I used scponly-4.6-3.el3.rf.i386.rpm and rsync-2.6.8-1.el3.rf.i386.rpm, the packages for Red Hat Enterprise Linux 3, since the Service Console is derived from RHEL 3.
2. As root, first install the rsync package with the rpm command.  Something like this should work:

   rpm -Uvh rsync-2.6.8-1.el3.rf.i386.rpm

3. After installing rsync, install scponly:

   rpm -Uvh scponly-4.6-3.el3.rf.i386.rpm

4. Edit the /etc/shells command to include the scponly shell, which in my testing was installed to /usr/bin.  You can simply append /usr/bin/scponly to the end of /etc/shells.

At this point, you should be able to create a user and set the user’s shell to scponly.  This will allow the user to use an SCP/SFTP client to transfer files, but it will not allow interactive shell access.

To test this, I used three file transfer methods:

• Interarchy, a Mac OS X application, via the SFTP protocol
• Command-line SCP from my Mac
• WinSCP on Windows XP, via both SFTP and SCP

In all three cases, I was able to successfully authenticate and transfer files, but I was not able to get access to the shell via SSH.  I would imagine that a determined individual could probably get by scponly, but in this instance we are only using it to provide a layer of protection for the shell.  For that function, it works well.

Interarchy is a well-respected FTP/SFTP client for Mac OS X, although the feature set now makes calling it an “FTP/SFTP client” a bit of a misnomer.  After, when you add in HTTP, WebDAV, and Amazon S3 support, it’s not exactly just an FTP/SFTP client any longer.  (For simplicity’s sake, we’ll continue calling it an FTP/SFTP client.)  For users of other FTP/SFTP clients, however, Interarchy’s interface is different enough to throw users off and prevent them from really being able to take advantage of the application’s full functionality.

Now that I’ve been using it for a couple of weeks, though, I’m starting to get the hang of the application, and beginning to understand (what I think) are the concepts behind the application.  In the event that other new users of Interarchy may find it useful, here are my thoughts.  I welcome additions, clarifications, or corrections from users more experienced than myself.

Multiple Windows

One difference that seems to throw new users of Interarchy is the use of multiple windows.  This didn’t really throw me at first, as I naturally use lots of windows anyway.  The idea here is that instead of using a two-paned “Midnight Commander”-type interface, Interarchy presents different windows for different remote locations, just like the Finder can present different windows for different file system locations.  In this respect, Interarchy is more Mac-like than most of the other similar applications out there.  (Side note: If you don’t like multiple windows, you can use multiple tabs in a single window instead.)  If you’ve ever used Linux with KDE or GNOME, then Interarchy’s windows with the URL address bar will look very familiar to you.

In addition, Interarchy has windows to report the status of mirror tasks, a transfers window, a transcript window, and of course the default Bookmarks window.

Bookmarks

Interarchy is built around the idea of bookmarks.  “Bookmark” isn’t the best name for this concept, but I can’t think of anything better either.  These bookmarks combine a set of basic building blocks (more on that in a moment) to create the desired result.  This makes Interarchy’s bookmarks much more than the bookmarks of a web browser; indeed, these bookmarks are more like Automator workflows.  (Hey, that’s a really good analogy.)

You’ll see this analogy carried throughout the application.  The “Connect to Server…” window, in which you can define “on-the-fly” bookmarks, is itself listed as a bookmark (although it’s a special bookmark that uses the “interarchy:” URL type).

Basic Building Blocks

Bookmarks in Interarchy combine three basic elements:  protocol, action, and parameters.

• Protocol:  Here you define the network protocol that should be used, such as FTP, SFTP, Amazon S3, HTTP, or WebDAV.  It’s kind of handy that the application and its bookmarks present a consistent interface regardless of the protocol(s) being used.
• Action:  Once the protocol is selected, you must then select the action. Actions include List (equivalent to a “traditional” connection that shows the contents of the selected server), Download, Upload, Mirror, New Net Disk, New Auto Upload, View, Edit, and others.  List is probably the one you’ll use the most.
• Parameters:  These will vary between actions.  For the List action, all you really need is the server name, path, username, and password.  For the Upload action, you’ll need those four parameters plus a source folder.

Mixing and matching these three building blocks allows you to build bookmarks (custom workflows) to do exactly what you want to do.

Here’s an example.  I maintain the web site for my church.  It’s not much (I’m not an HTML programmer by any stretch of the imagination), but God honors the effort anyway.  I have three bookmarks defined to help manage the church web site:

1. A List bookmark, which allows me to navigate the filesystem where the web site’s files are stored, edit files, etc.
2. A Mirror Download bookmark, which downloads an exact copy of the web site to my MacBook Pro.
3. A Mirror Upload bookmark, which uploads the local copy of the web site to the remote server.

Using these bookmarks, I can quickly and easily download a mirror copy of the web site, edit it locally, then publish it back up to the server with just a few clicks.  (Yes, I could probably do this easier with a straight-up Mirror task, but I’m not quite ready to go there yet.  I like control.)  Other applications have the same kind of synchronization functionality (Cyberduck does), but they require you to first establish a connection to the server, then initiate the synchronization.  Interarchy allows you to just do the task in one step.

Hopefully, you’ve found this information at least a little but useful.  I’m sure that as I continue to use Interarchy, I’ll continue to find new ways of streamlining these types of tasks.  Anyone have any tips they care to share?

Unable to cope with the slow performance of Cyberduck any longer (even though I love everything else about the application), I started looking at other Mac FTP/SFTP clients.  Since performance was the key driving factor causing me to seek a new client, I thought it would probably be important to perform some informal performance comparisons between the major candidates—Interarchy, Transmit, and Fetch (and, for reference, Cyberduck as well).

First, the parameters of the test:

• The source system was, of course, a Mac.  Specifically, a MacBook Pro running Mac OS X 10.4.8 with all available and applicable updates installed.
• The destination system was a server running ESX Server 3.0.1.  It looks like ESX Server 3.0.1 uses OpenSSH 3.6.1p2.
• I transferred an ISO of Windows Server 2003 R2 x64; the file was about 592MB in size.  I deleted the file from the destination server after each transfer.

The results of the test were as follows:

Transmit 3.5.5: 39 seconds
Fetch 5.2: 39 seconds
Interarchy 8.2.2: 29 seconds
Cyberduck 2.7.2: 9 minutes 53 seconds

I hadn’t truly realized just how much slower Cyberduck was until I ran this test.  The difference is quite dramatic.  As you can see, the test results between Transmit, Fetch, and Interarchy were pretty close; Interarchy edged the others out but only by a small margin.

Given that the performance is pretty much the same between the three replacement candidates comes down to aesthetics and features.  I love the Kerberos support in Fetch (I’m a Kerberos fan—in case you hadn’t noticed by all the articles I write about leveraging the Kerberos support in Active Directory for cross-platform integration), but really don’t like the interface.  Likewise, Transmit is nice, has a Quicksilver plug-in and a Dashboard widget, Keychain support, Spotlight integration, etc., but something about the interface doesn’t seem to fit.  I’m not sure what it is.  This is not a knock against Transmit or Panic; I paid for and use Unison, Panic’s Usenet/NNTP client.

Interarchy’s interface has its quirks, but I suppose I’m getting used to them now because I don’t notice them as much.  I just found and installed the Interarchy Quicksilver plug-in yesterday, which solves one complaint, but I still wish Interarchy would expand its Growl support.

Each of the apps has its advantages and disadvantages, as you can see.

What would be the ideal solution?  The ideal solution would be a way to tune Cyberduck’s performance so that it was in the same ballpark as the other applications (not necessarily faster, just in the same general area).  I don’t suppose anyone has any performance tuning tips for Cyberduck?

I’d gotten turned on to Cyberduck as my primary FTP/SFTP client after really getting into Growl, the global notification system for Mac OS X.  The application I was using at the time, Fugu, didn’t have Growl support.  Cyberduck did, so I switched, and I’ve been using Cyberduck ever since.

I like the Cyberduck interface; it seems to make sense to me and I’ve never really run into any major compatibility issues (seems like I ran into one minor problem after an upgrade of OpenSSH on one of my servers, but that problem was quickly resolved as I recall).  The Growl support is, of course, excellent, and Cyberduck also offers a veritable laundry list of features—integrated support for Spotlight, a Dashboard widget, Keychain support, multiple windows, etc.  It even comes as a Universal binary.  (The features are far too many to list here; refer to the Cyberduck web site for complete information.)

Sound like a great application?  It is—if you don’t need to transfer large files.  Since I started out just using Cyberduck to move some small web pages back and forth to my web server, these were mostly small files and I didn’t really notice any performance hit.  Sure, it seemed a bit slower than command-line SFTP or SCP and it seemed to be a bit of a memory hog, but I figured it was just GUI overhead and thought no more about it.  For what I was doing at the time, it worked fine.

Recently, though, I’ve been needing to transfer much larger files to and from some SFTP servers on my local LAN.  How large?  ISO images ranging from 300MB to 600MB, sometimes multiple ISO images at a time.  Generally, the file transfers will complete, but they are just plain slow.  Almost painfully slow.  So slow, in fact, that I’ve been driven to looking at alternatives.

I’m currently evaluating Interarchy.  While the interface is a bit quirky (although I suppose that is due to being predisposed to an interface like Cyberduck’s), the performance is astounding.  I can transfer multiple ISO images in minutes, not hours as with Cyberduck.  It’s almost unbelievable.

I have yet to decide whether I’ll just buy Interarchy or if I’ll evaluate two other potential candidates, Transmit and Fetch.  Both applications have gotten good reviews, but—being the UI stickler that I am—neither of them sports as modern a UI as Interarchy (I really like the unified toolbar look).

My primary complaint with Interarchy is the price.  Sixty bucks seems a bit high for this type of application; both Transmit and Fetch (other options to replace Cyberduck) charge about half that.  Of course, the other applications don’t offer the same set of functionality that Interarchy offers, either.  But will I actually use that functionality?  Amazon S3 support is great, but will I really use Amazon S3?  I don’t have a WebDAV server, so is it worth paying for WebDAV support?  Is it worth paying for network tools that duplicate functionality already in the base operating system?

What do you think?  If you are a Fetch, Transmit, Interarchy, Fugu, or even Cyberduck user, please post in the comments and tell me what you think.