Tags

After figuring out yesterday how to list all the unique MailTags projects in use, today I worked out how to list all the unique MailTags keywords in use.  As it turns out, it wasn’t as hard as I had suspected that it would be.

(Note to experienced UNIX hackers:  I’m sure that I probably could have worked out a more elegant solution involving more pipes and redirection, but this works for a UNIX apprentice such as me.)

Here are the commands I used to parse down the list of unique keywords (this command is broken across two lines, but should be typed all on one line):

mdls <path to mailbox> | grep kMDItemKeywords |
awk ‘{print $3 $4 $5 $6 $7 $8 $9}’ >> outputfile.txt

This uses the mdls command to list all the metadata from the messages in the specified directory, pipes that through grep to pick out only the lines containing “kMDItemKeywords”, then uses awk to list all the keywords (which are in a comma-delimited list surrounded by parentheses.  You’ll want to repeat this command for every mailbox you use (I have different mailboxes to archive messages by year, so I had to repeat this for 2005, 2006, etc.)

Once we have the keywords list, then we munge it:

cat outputfile.txt | tr ‘,’ ‘\n’ > outputfile2.txt
sed s/\(//g < outputfile2.txt > outputfile3.txt
sed s/\)//g < outputfile2.txt > outputfile3.txt
sort outputfile3.txt > sorted-file.txt
uniq sorted-file.txt > unique-file.txt

This replaces commas with a newline (the tr command), strips out the parentheses (the next two sed commands), then sorts it and returns only the unique values.  The end result, stored in unique-file.txt, contains a list of unique MailTags keywords used in all your mailboxes.

Useful, eh?

Spurred to see if I could become more efficient than I am currently, I’ve been experimenting with some GTD-style applications.  As a result of my experiments, I found that I needed a list of all the MailTags projects I’ve ever used in any of my mailboxes.  (To make a long story short, I want to use the same project names across all applications—Mail, GTD application, Spotlight comments for files, etc.)  While you can see a list of projects in the Preferences, that list does not include all projects that might ever have been used.

It only took a few minutes of experimentation in the Terminal to come across the mdls command, which lists metadata for a file (or group of files).  Perfect!  Using mdls, I listed the metadata for an e-mail message, and found that it was the kMDItemProjects attribute that was used by MailTags.  With that information in hand, I hacked together this quick process for listing all the projects in use across all my mailboxes (I’ve used a backslash to indicate line continuation; lines are wrapped here for readability):

mdls <mailbox path>/*.emlx | grep kMDItemProjects >> \
~/all-projects.txt

Repeat this command for each mailbox (for example, I have a separate archive mailbox for each year).  Once you’ve enumerated the information for all the mailboxes, then parse it down like this:

sort all-projects.txt > sorted-projects.txt
uniq sorted-projects.txt > unique-projects.txt

Now the file “unique-projects.txt” contains all the unique project names that are in use across all your mailboxes.  Very handy!  Now, if only obtaining a unique list of all the MailTags keywords were as easy…

This is really exciting news.  Development on Cocoalicious, the Mac OS X native application that front-ends del.icio.us, has started back up again.

I’ve blogged many times about Cocoalicious (starting as far back as June of 2005) and how much I enjoy using the application to manage my del.icio.us bookmarks.  I was really disappointed that development had stalled, and had even started searching for replacements to the application.  Fortunately, it looks like the new developer (who is working with the original author, not replacing him, from what I understand) is already seeking feedback and ideas for future versions.

Personally, I’m pretty thrilled with the application as it is, and have only one feature request:  please, please, PLEASE drop the brushed metal interface.  Or at least offer us an option to toggle back and forth.  I’d love to see a fresh new UI like that used by Mail.app or NetNewsWire, with the tags in a pane on the left and your bookmarks listed on the right, and a divider (like the one used now) to open, close, or resize the built-in browser.  Combine that with a new, modern unified toolbar (not Mail.app’s lozenges, please!) and perhaps incorporate some of the tag UIs that have been proposed (like this one), and you’ve got yourself one killer del.icio.us client.

The idea behind del.icio.us is great, but for me it becomes truly useful using a “rich client” instead of a web browser.  For a long time, Cocoalicious has been that “rich client,” offering a combination of native Mac OS X technologies with the web services offered by del.icio.us.  Unfortunately, it appears as though Cocoalicious is no longer under active development, and so I’ve gone seeking other solutions.

There are quite a few bookmark managers out there for the Mac, but not so many that offer integration with del.icio.us.  Likewise, there are a number of utilities that offer to make posting to del.icio.us easier (Pukka and Postr come to mind) but don’t necessarily offer the bookmark management functionality upon which I rely.  So far, I’ve only found two applications that have the right balance of functionality.

The first of these is WebnoteHappy.  It looks as if WebnoteHappy originally started out as “just” a bookmark manager; del.icio.us support seems to be an add-on rather than an integral part of the application itself.  Nevertheless, WebnoteHappy does have a couple of things going for it:

• It supports integration with NetNewsWire, my RSS reader, so that I can post URLs directly from NNW’s context menu.  (Currently, only Cocoalicious, Pukka, Postr, and WebnoteHappy appear to be supported.)
• It supports AppleScript.
• It supports Smart Folders to group bookmarks according to tags, description, or notes.

The best part of del.icio.us, to me, is the tags.  This is where WebnoteHappy seems to be the weakest.  I can’t browse my bookmarks by tags (although I could create a Smart Folder based on tags), there’s no tag autocompletion, and when posting to del.icio.us via WebnoteHappy from NNW I’m not given the option to assign any tags (indeed, I’m not even given the option to share the bookmark via del.icio.us).

The second application is a relatively new application; it’s called Socialist.  Socialist appears to be built from the ground up to be a “rich” del.icio.us client.  The relative immaturity of Socialist is showing up in some areas, though:

• No AppleScript support.
• No integration with NNW.  (Granted, the list of supported applications is fairly small, but this is a feature I use regularly.)

Fortunately, Socialist does support tags, and does provide a way to browse bookmarks via tags.  The current release doesn’t support browsing via multiple tags or tag autocompletion, but supposedly those features are in the next version of the software (which is due out soon).

Each application has its own unique strengths and weaknesses, and both are lacking some features that I would love to see:

• Growl support (to provide a Growl notification when a URL is successfully posted to del.icio.us)
• AppleScript support (so URL management tasks can be automated a bit more)
• Spotlight integration (ability to search URL and note text from the Spotlight menu)

Of course, I already mentioned browsing via tags (including the ability to select multiple tags and see only the bookmarks tagged with all the selected tags) and tag autocompletion.  If NNW integration isn’t possible, then the ability to at least pull the contents of the clipboard into the new bookmark sheets in each application would be good.  An entry on the Services menu would be handy as well.

Any other products out there I should be considering?  Anyone have any feedback on one of these two products?  I’d love to hear from real-world users on what they like or don’t like about either of these two applications.

My rudimentary “document management system” is comprised of a variety of technologies, some built into Mac OS X and some added on via separate third-party applications.  At the core of the system is Spotlight, the search engine that is the object of a love-hate relationship with many Mac users.  Despite Spotlight’s shortcomings, it is still a very useful tool to have.  (I will have to admit that I am sure I am still not taking full advantage of Spotlight’s abilities.)

On top of Spotlight I’ve added a number of pieces:

• Spotlight comments in Finder:  I place relevant information in the Spotlight comments in Finder so that the Spotlight engine can find and index these comments.  Keywords/tags are separated by spaces and surrounded by brackets (i.e., “[Macintosh Security]”) and additional comments—such as a brief description of the document—follow immediately thereafter.
• Project-specific keywords/tags:  I have project-specific keywords/tags for the major customer projects in which I am involved.  For example, if I were handling an Exchange migration for XYZ Widgets, my project-specific keyword would be “Proj:XYZ-ExMig”.  This makes it incredibly easy to locate documents relating to a specific document.
• MailTags:  MailTags, a handy add-on for Mail, lets me apply some of the same metadata to my e-mail messages as to my file system.  Then, when I assign a particular message to a project (using the same naming convention as my project-specific keywords mentioned above), I can search for both project-related documents and project-related messages with a single Spotlight search.
• Keywords/Tags in Address Book and iCal:  While not specifically related to document management per se, using the same keywords/tags on Address Book contacts and iCal appointments further unifies the structure and makes it easier to see all related resources.  In addition, it also makes it easy to use the Spotlight system service to find related documents when viewing a contact or an appointment.
• Automator workflow for adding Spotlight comments:  To help streamline the process of adding these comments to files in Finder, I’ve added an Automator workflow that prompts me for the text to add to the Spotlight comments and then either appends that text to the existing comments or replaces the existing comments with what I specify.  This makes it easy to tag large numbers of documents at the same time.
• Microsoft Office metadata:  I also fill out the metadata for all Office documents (via File > Properties).  This is something I’ve been doing for years (since the introduction of the “Find Files” command in Word for Windows 2.0, circa 1994-1995), so it’s very natural for me.  All of the Office applications are configured to automatically prompt me for document properties when I save a new document.  In the document properties, I specify the same keywords/tags as in the Spotlight comments as well as author, manager, title, description, etc.
• Saved searches:  To help group together documents, I use a small selection of saved searches—aka Smart Folders—based on raw Spotlight queries.  Using raw Spotlight queries allows a finer level of control over what items are returned by the search.  For example, the following Spotlight query shows me only active project-related documents:

  (kMDItemFinderComment =*Proj*) && (kMDItemFinderComment != *Inactive*)

  I’m still fine-tuning my use of raw Spotlight queries, trying to make sure that I stick to indexed metadata so that Spotlight doesn’t have to perform a real-time search everytime I open the Smart Folder.  I’d appreciate anyone who has some tips there to share them in the comments.

So, based on my own personal workflow and these additional tools, it’s easy for me to create a new project-related document (say, a migration plan or a network diagram), attach the associated metadata (either manually or via the Automator workflow), and have easy and quick access to the document afterward, without really having to pay too much attention to where I saved the document.

I do still use regular old folders (directories) in Finder as well; I’m not sure if that’s a holdout of the “old school” way of doing things, but I’m just not ready to completely give up traditional folders just yet.

Changes and/or additions that I’m considering adding include:

• Using Quicksilver’s “File Tags” plug-in:  As my use of Quicksilver has continued to increase (I’m hoping to now start using it to track and access my del.icio.us bookmarks, since Cocoalicious went belly up), I’ve also explored the use of Quicksilver for tagging.  My main concern with Quicksilver’s tagging module is the need for a prefix that is added to every tag.  Right now I don’t use any sort of prefix, and you can’t (unfortunately) tell it not to use a prefix.  Perhaps I’ll use Quicksilver to create a trigger for my Automator workflow…
• Changing keyword/tag format:  Partially because of Quicksilver (see previous bullet), I’m thinking of adding some sort of prefix to denote keywords/tags as such, instead of just part of the Spotlight comments or the document’s content.  The idea is that searching for “@Proj:XYZ-ExMig” would get only files tagged as such, not all items that may have that content inside them.  I still haven’t quite decided on this one.  Anyone have any thoughts?  Would it really be worth the effort?

So, there you have it…my homegrown document management system.  I’m open to suggestions for improvement or additional tools to make it more effective.  What am I missing that could make this better?

I’ve been wrestling with Technorati not indexing my weblog for quite a while now.  It appears that ever since I added Ultimate Tag Warrior to my weblog and moved the tags into the keywords portion of the weblog (instead of the posting body), Technorati stopped indexing the site.  It appears that the tags aren’t being added to the RSS and Atom feeds in the content, and therefore Technorati won’t properly index the site.

I could go back to putting the tags in the body of the posting, but I really don’t feel like it, and I like having them in the keywords portion of the posting.  Since the Technorati links weren’t returning any posts on my weblog, I’ve just removed the Technorati search links on the category and tag pages.  Perhaps that’s childish, or selfish, or whatever, but the links just don’t seem useful any more.

In addition, I’ve added links to search del.icio.us for related bookmarks when browsing the category pages, and I’ve added a link for subscribing to a category’s RSS feed.  Visitors now use the exact same links for both tags and categories to find related bookmarks and subscribe to the RSS feed.

Just for easy reference, each category’s RSS feed is available at the following URL:

http://blog.scottlowe.org/<category name>/feed

In addition, each tag’s RSS feed is available at the following URL:

http://blog.scottlowe.org/<tag name>/feed

By default, these URLs will return an RSS 2.0 feed.  To access an RSS 0.92 feed, add “/rss” to the end of each of these URLs.  There is no Atom feed for categories and tags just yet.  There is an Atom 1.0 feed for the overall weblog, though.

Since each category’s RSS feed is available from the category page, I’ve removed the links to the RSS feeds in the sidebar.

UPDATE:  I also added the plugin Tags in the Head to add tags as meta keywords in the header of each page.  This is a pretty handy plugin that also ties in nicely with Ultimate Tag Warrior.

Due to the changes to the site I described earlier, there are some changes ahead in how posts will be categorized and tagged.  Because Ultimate Tag Warrior now causes WordPress to include the tags as well as the category in the RSS feed (to properly link up with Technorati), I will be removing tags that duplicate the category and instead using tags that are more specific, where possible.

For example, posts placed into the Microsoft category won’t have the Microsoft tag also assigned to them; otherwise, the feed shows two entries for Microsoft—one for the category, and one for the tag.  Likewise, posts categorized into Security won’t be tagged Security as well; instead, I’ll break down the tags into Patches, Vulnerabilities, Exploits, etc.  I haven’t come up with the specific tags that I’ll use just yet, but I do want to continue to try to keep the tags I’m using here consistent with the tags I’m using on del.icio.us.

UPDATE:  I started to go back and update some of the older articles with this scheme, and then I realized that if I don’t tag the article with the same tag as the category, then site visitors can’t search for related weblog entries or bookmarks with those tags.  So, I’ve decided to leave the tags as they are and not worry about duplicate tags/categories.

I wouldn’t dare to describe myself an expert on the use of such sites or services as del.icio.us, Technorati, or other sites that promote the use of “tags” as a means of organizing links or other content.  I do use these sites (you’ll note the “Technorati Tags” added to each post), and I do find them very useful.  However, I’m beginning to shift tactics in my use of tags, and hoping that this change will make these services even more useful.

Up until now, I’ve used tags primarily as categories, as a way of grouping links (or other content) together.  For example, my Linux tag on del.icio.us contains links that all primarily center around Linux.  That makes perfect sense, but what about links that discuss Linux integration with Windows?  With my old system, that creates a quandary, a conflict that is not necessarily easy to resolve.

Moving forward, I’m going to start using tags more as keywords than categories.  This will mean that more items will have multiple tags associated with them (and that makes sense), but it will also mean more tags than I have used in the past.  This will allow me to more easily tag links and content in such a way that I hope will be more useful to myself and others.

I will continue, however, to attempt to use the same sets of tags across multiple sites and services, just as I do now with my del.icio.us links and my weblog content.

If anyone has any feedback on other ways I can make tagging work more effectively, I’d love to hear about it—perhaps someone out there would care to share their tips, tricks, and techniques?