Infinite Loop / The Apple Ecosystem

Mac OS X 10.4 Tiger


Metadata revisited

It's been a while since I've written about metadata in Mac OS X. First, a few brief definitions for newcomers to the topic.

  • Colloquially, metadata is "data about data."
  • File metadata is information associated with a file that is not logically part of the file's actual content (i.e., its data), although it may be physically.
  • File system metadata is file metadata that is stored as part of the file system data structures dedicated to each file. A file's modification date is a common example of file system metadata, but the Mac has historically supported much more: type and creator codes, Finder flags, icon position, label, etc.

All of these terms are subtly different and are generally not interchangeable. I will try to use them very precisely in this article.

So enamored am I with the world of file metadata that I wrote an entire article about it back in the summer of 2001. In the early days of Mac OS X's development, it was difficult to divine Apple's intentions in this area. The file metadata mechanisms in the developer versions and early public releases of Mac OS X seemed to be more a product of their mixed Apple/NeXT heritage than a manifestation of a coherent plan.

But even several months before the Public Beta release, I was already growing somewhat concerned about the apparent intentions of the mothership. Reviewing what I wrote in June of 2000, my trepidation is clear. At that time, the only official Apple statement on the topic was that they "strongly encourage developers to use file extensions as alternative means for identifying document types."

The arrival of an actual 10.0 release on March 24th of 2001 was such a significant event that the metadata issue was swept away by larger concerns. "Will this transition work?" "Can I run my old apps?" "What the heck is the deal with the performance?"

But by the time version 10.1 was released in October of 2001, Apple's direction was clear. The decree was simple: all files created by Mac OS X applications must include a file name extension. (Cue dramatic music.) My worst fears were realized, and my reaction was swift and merciless.

I didn't stop there. I went on to formally propose an alternate course of action for Apple. I created an online petition (stop laughing) in support of the proposal that gathered almost 10,000 signatures and comments from a broad cross-section of the Mac community. Here's my favorite.

I'm confident that the folks at Apple will eventually recognize the role that basic metadata, such as type/creator, plays in the Macintosh user experience. When I was working on the Finder in 1982 and designed the type/creator, desktop database, and resource mechanisms, I wanted to make sure that the user would almost never have to care about what the file's type was, how it should be named, or what application was appropriate to edit it. At the same time, I felt that the Finder should allow users to override the Finder's application preference as needed; thus, drag-and-drop to an application was provided. In the following years the software designers at Apple have done beautiful work in maintaining this ease of use with the Finder and File Exchange. Promoting a least-common-denominator solution is inappropriate for a company sustained by innovation; Apple needs to embrace the entire concept of metadata for user information, and consider type/creator as the beginning of something much more powerful for OSX.

Bruce Horn
Member of the original Macintosh development team.
Signature 252.

I filed the metadata proposal as an official bug (2826368) in Apple's bug tracking system and waited for what I hoped would be some sort of change in direction from Apple—or at least an acknowledgment of the issue.

In early 2002, Apple hired Dominic Giampaolo, the creator of the Be File System. BFS was one of the most advanced personal computer file systems of its time, with support for arbitrarily extensible file system metadata and efficient, real-time search queries. Dominic wrote an excellent book about BFS, and file system design in general. The book is out of print now, but it's available as a free download. Even today, BFS's features and performance are not matched by many commonly used file systems.

Dominic's arrival at Apple was seen as a sign of things to come. It doesn't take much to get Mac enthusiasts excited about future possibilities. Fair or not, expectations were high.

In August of 2002, Mac OS X 10.2 Jaguar was released with no significant changes to the file metadata policies or mechanisms. Two days after the release of Jaguar, Apple closed my metadata proposal bug in its bug tracking system, assigning it the status message "Closed/Behaves correctly."

This put quite a damper on the whole issue of Mac OS X metadata. The irrational Giampaolo mania of the Mac metadata prognosticators waned considerably. Having said everything I had to say on the topic, and having done everything I thought I could to effect some sort of change inside Apple, I largely abandoned the issue as well. What more could be done? Objections had been voiced, advice given, and file system gurus hired. All we could do was wait.

Mac OS X 10.3 Panther arrived in October of 2003. It contained no significant changes to the file metadata policies or mechanisms. (Noticing a pattern here?) So we waited some more.

The year 2004 arrived without the expectation of a new major version of Mac OS X until at least early 2005. Anyone who still managed to maintain an interest in the topic had to face a grim reality: Despite all efforts and expectations, the file metadata situation in Mac OS X had not significantly changed since version 10.1 in 2001.

A new hope

In March of 2004, Apple quietly released an update to its HFS Plus Volume Format technote (TN1150). The update described a new variant of HFS+ called HFSX.

HFSX is an extension to HFS Plus to allow additional features that are incompatible with HFS Plus. The only such feature currently defined is case-sensitive filenames.

Case-sensitive HFS+ (as it is called in end-user interfaces like Disk Utility) was actually introduced in Mac OS X 10.3 Panther, but was only exposed in the GUI in Mac OS X Server. Still, the new name and the stated intent to "allow additional features that are incompatible with HFS Plus" showed that Apple was willing to do something in the realm of file system technology.

Then there was the section named "Future Support for Named Forks" which contained these refreshingly non-archaic notions about file system metadata.

A number of products have implemented special-purpose solutions for storing their file- and directory-related data. But because these are not managed by the file system, they can become inconsistent with the file and directory structure.

HFS Plus has an attribute file, another B-tree, that can be used to store additional information for a file or directory. Since it is part of the volume format, this information can be kept with the file or directory as it is moved or renamed, and can be deleted when the file or directory is deleted.

Then things started to really heat up.

The contents of the attribute file's records have not been fully defined yet, but the goal is to provide an arbitrary number of forks, identified by Unicode names, for any file or directory.

(Emphasis added.) Now granted, it's been widely known for many years that HFS+ could technically support an arbitrary number of named forks, but the language in the technote has slowly shifted in its views towards these capabilities. After the HFSX update, the intent seemed very clear.

Still, alternate data streams ("forks" in Mac parlance) have always seemed like a very heavyweight implementation of arbitrarily extensible metadata. This is another reason why HFS+'s ability to support multiple named forks hadn't been much more than a curiosity in the past. Although the technote language changes renewed some interest, the technical issues seemed to remain.

But there was one final tidbit of information in the technote. Although it states that "the contents of the attribute file's records have not been fully defined yet," there was enough documentation to get an idea of what was to come.

Each [attributes file] record starts with a recordType field, which describes the type of attribute data record. The recordType field contains one of the following values.

enum {
    kHFSPlusAttrInlineData  = 0x10,
    kHFSPlusAttrForkData    = 0x20,
    kHFSPlusAttrExtents     = 0x30

The values have the following meaning:

Reserved for future use.
This record is a fork data attribute. You can use the HFSPlusAttrForkData type to interpret the data.
This record is an extension attribute. You can use the HFSPlusAttrExtents type to interpret the data. A record of type kHFSPlusAttrExtents is really just overflow extents for a corresponding record of type kHFSPlusAttrForkData.

(Again, emphasis added.) Inline attributes. Now we were getting somewhere. While entire forks seemed like overkill for small pieces of data, the word "inline" sounded suitably lightweight. The phrase "reserved for future use" increased the metadata expectations for Tiger.

As it turns out, those expectations would be met in some very interesting ways. Starting in mid-2004, Apple began to publicly demonstrate and market some significant new metadata-related features scheduled for Tiger. If you're a casual Mac user, or maybe not even a Mac user at all, you may think you already know where this is going. Think again.

Expand full story

You must to comment.


You May Also Like