Smart Folders Suck Ass

Rarely have I seen an interface or feature as useless or just downright broken as Smart Folders in the Mac OS X 10.4 Finder. Thought they might be handy, so I tried to make one. Here's what happened.

My goal was to create a Smart Folder that contained all my Final Cut Pro project files from the past year. In the Finder I clicked "command-f" to perform a search. A new window opened with some default search criteria pre-filled. I set the search location to "Home" and then set the "Kind" to "Others..." Next, I attempted to set up my "Kind" criteria to search for Final Cut Pro project files. Clicking the little arrow next to the text entry field brings up a list of document types. The list would seem to be rather exhaustive. It's huge. But the bad thing is that it seems to continually update, so as you begin to scroll through the list, it keeps jumping back to the top as the list is updated. Type-ahead find is also enabled, but again, even using type-ahead the list still jumps to the top. Too much typing and you'll get the beachball. All this makes the list of document types nearly unusable, or at least unreadable. Unless you know the exact name of the document type you're looking for, you'll be lucky to find it in this horribly broken drop-down. Fortunately I was able to correctly guess that the document type I was looking for was called "Final Cut Pro Project File." After typing this in, I got a list of all the FCP files in my home account. Okay. Great. Halfway to my goal, I saved the Smart Folder to the "Saved Searches" location and added it to my Sidebar (there are only three possible save locations for Smart Folders, by the way: Saved Searches, Desktop and Home).

My New Smart Folder: Let the Ass-Sucking Begin
(click image for larger view)

So, I had a Smart Folder that listed all my FCP files, but I wanted to narrow this a bit to include only those FCP files from the last year. No problem, right? I opened my Smart Folder and clicked the Edit button. Again I was presented with search criteria at the top of my window. For my second set of criteria I selected "Created Within Last 1 Years." At this point the Finder beachballed for about twenty seconds, then restarted. Fortunately — and I use the term loosely — when it came back to life it had actually preserved my Smart Folder and I was able to save it.

So, finally, after much effort, I had my Smart Folder of all FCP documents in my home account, created in the last year. Neat-O! I decided to see what I could do with this list. So I opened up my Smart Folder, and started poking around. The first thing I noticed was that the list was ordered alphabetically. There appeared no obvious way to reorder the list. It was just a flat list of files, and a long one at that. It didn't even show labels.

The Default Smart Folder View: Utterly Useless
(click image for larger view)

Fortunately, as it happens, switching to List view in the Finder window yielded a more useful view of my data, but it took me a bit of time to even realize I could do this.

The View Switch: Good Thing I Remembered This
(click image for larger view)

It's a little confusing that the default view in a Smart Folder is really quite limited in its usefulness. Why do they use the Spotlight view for this? The standard list view is just as user-friendly and a thousand times more useful. It's hard to believe this is an Apple-designed interface.

List View: Finally Something Vaguely Useful
(click image for larger view)

In the end, I'm not sure how useful my Smart Folder will be. Honestly, I've rarely found myself wishing I had this sort of functionality, and the confounding and oftentimes irritating user experience that is Smart Folder configuration doesn't really leave me with much desire to come up with a use for it. Nor does it instill much confidence that this view of my data is accurate. If Smart Folders can cause the Finder to crash, can I really trust them to accurately perform their searches?

Smart Folders are a neat idea. But I really think they're just one more area where Spotlight needs a ton of work before they can really be of much use. I recently read a description of Smart Folders as "underused." Yeah, well, maybe that's less because no one knows about them and more because they just plain ol' suck ass.

Backing Up with RsyncX

In an earlier post I talked generally about my backup procedure for large amounts of data. In the post I discussed using RsyncX to back up staff Work drives over a network, as well as my own personal Work drive data, to a spare hard drive. Today I'd like to get a bit more specific.

Installing RsyncX

I do not use, nor do I recommend the version of rsync that ships with Mac OS X 10.4. I've found it, in my own personal tests, to be extremely unreliable, and unreliability is the last thing you want in a backup program. Instead I use — and have been using without issue for years now — RsyncX. RsyncX is a GUI wrapper for a custom-built version of the rsync command that's made to properly deal with HFS+ resource forks. So the first thing you need to do is get RsyncX, which you can do here. To install RsyncX, simply run the installer. This will place the resource-fork-aware version of rsync in /usr/local/bin/. If all you want to do is run rsync from the RsyncX GUI, then you're done, but if you want to run it non-interactively from the command-line — which ultimately we do — you should put the newly installed rsync command in the standard location, which is /usr/bin/.¹ Before you do this, it's always a good idea to make a backup of the OS X version. So:

sudo cp /usr/bin/rsync /usr/bin/rsync-ORIG

sudo cp /usr/local/bin/rsync /usr/bin/rsync

Ah! Much better! Okay. We're ready to roll with local backups.²

Local Backups

Creating local backups with rsync is pretty straightforward. The RsyncX version of the command acts almost exactly like the standard *NIX version, except that it has an option to preserve HFS+ resource forks. This option must be provided if you're interested in preserving said resource forks. Let's take a look at a simple rsync command:

/usr/bin/rsync -a -vv /Volumes/Work/ /Volumes/Backup --eahfs

This command will backup the contents of the Work volume to another volume called Backup. The -a flag stands for "archive" and will simply backup everything that's changed while leaving files that may have been deleted from the source. It's usually what you want. The -vv flag specifies "verbosity" and will print what rsync is doing to standard output. The level of verbosity is variable, so "-v" will give you only basic information, "-vvvv" will give you everything it can. I like "-vv." That's just the right amount of info for me. The next two entries are the source and target directories, Work and Backup. The --eahfs flag is used to tell rsync that you want to preserve resource forks. It only exists in the RsyncX version. Finally, pay close attention to the trailing slash in your source and target paths. The source path contains a trailing slash — meaning we want the command to act on the drive's contents, not the drive itself — whereas the target path contains no trailing slash. Without the trailing slash on the source, a folder called "Work" will be created inside the WorkBackup drive. This trailing slash behavior is standard in *NIX, but it's important to be aware of when writing rsync commands.

That's pretty much it for simple local backups. There are numerous other options to choose from, and you can find out about them by reading the rsync man page.

Network Backups

One of the great things about rsync is its ability to perform operations over a network. This is a big reason I use it at work to back up staff machines. The rsync command can perform network backups over a variety of protocols, most notably SSH. It also can reduce the network traffic these backups require by only copying the changes to files, rather than whole changed files, as well as using compression for network data transfers.

The version of rsync used by the host machine and the client machine must match exactly. So before we proceed, copy rsync to its default location on your client machine. You may want to back up the Mac OS X version on your client as well. If you have root on both machines you can do this remotely on the command line:

ssh -t root@mac01.systemsboy.com 'cp /usr/bin/rsync /usr/bin/rsync-ORIG'

scp /usr/bin/rsync root@mac01.systemsboy.com:/usr/bin/

Backing up over the network isn't too much different or harder than backing up locally. There are just a few more flags you need to supply. But the basic idea is the same. Here's an example:

/usr/bin/rsync -az -vv -e SSH mac01.systemsboy.com:/Volumes/Work/ /Volumes/Backups/mac01 --eahfs

This is pretty similar to our local command. The -a flag is still there, and we've added the -z flag as well, which specifies to use compression for the data (to ease network traffic). We now also have an -e flag which tells rsync that we're running over a network, and an SSH option that specifies the protocol to use for this network connection. Next we have the source, as usual, but this time our source is a computer on our network, which we specify just like we would with any SSH connection — hostname:/Path/To/Volume. Finally, we have the --eahfs flag for preserving resource forks. The easiest thing to do here is to run this as root (either directly or with sudo), which will allow you to sync data owned by users other than yourself.

Unattended Network Backups

Running backups over the network can also be

completely automated and can run transparently in the background even on systems where no user is logged in to the Mac OS X GUI. Doing this over SSH, of course, requires an SSH connection that does not interactively prompt for a password. This can be accomplished by establishing authorized key pairs between host and client. The best resource I've found for learning how to do this is Mike Bombich's page on the subject. He does a better job explaining it than I ever could, so I'll just direct you there for setting up SSH authentication keys. Incidentally, that article is written with rsync in mind, so there are lots of good rsync resources there as well. Go read it now, if you haven't already. Then come back here and I'll tell you what I do.

I'd like to note, at this point, that enabling SSH authentication keys, root accounts and unattended SSH access is a minor security risk. Bombich discusses this on his page to some extent, and I want to reiterate it here. Suffice to say, I would only use this procedure on a trusted, firewalled (or at least NATed) network. Please bear this in mind if you proceed with the following steps. If you're uncomfortable with any of this, or don't fully understand the implications, skip it and stick with local backups, or just run rsync over the network by hand and provide passwords as needed. But this is what I do on our network. It works, and it's not terribly insecure.

Okay, once you have authentication keys set up, you should be able to log into your client machine from your server, as root, without being prompted for a password. If you can't, reread the Bombich article and try again until you get it working. Otherwise, unattended backups will fail. Got it? Great!

I enable the root account on both the host and client systems, which can be done with the NetInfo Manger application in /Applications/Utilities/. I do this because I'm backing up data that is not owned by my admin account, and using root gives me the unfettered access I need. Depending on your situation, this may or may not be necessary. For the following steps, though, it will simplify things immensely if you are root:

su - root

Now, as root, we can run our rsync command, minus the verbosity, since we'll be doing this unattended, and if the keys are set up properly, we should never be prompted for a password:

/usr/bin/rsync -az -e SSH mac01.systemsboy.com:/Volumes/Work/ /Volumes/Backups/mac01 --eahfs

This command can be run either directly from cron on a periodic basis, or it can be placed in a cron-run script. For instance, I have a script that pipes verbose output to a log of all rsync activity for each staff machine I back up. This is handy to check for errors and whatnot, every so often, or if there's ever a problem. Also, my rsync commands are getting a bit unwieldy (as they tend to do) for direct inclusion in a crontab, so having the scripts keeps my crontab clean and readable. Here's a variant, for instance, that directs the output of rsync to a text file, and that uses an exclude flag to prevent certain folders from being backed up:

/usr/bin/rsync -az -vv -e SSH --exclude "Archive" mac01.systemsboy.com:/Volumes/Work/ /Volumes/Backups/mac01 --eahfs > ~/Log/mac01-backup-log.txt

This exclusion flag will prevent backup of anything called "Archive" on the top level of mac01's Work drive. Exclusion in rsync is relative to the source directory being synced. For instance, if I wanted to exclude a folder called "Do Not Backup" inside the "Archive" folder on mac01's Work drive, my rsync command would look like this:

/usr/bin/rsync -az -vv -e SSH --exclude "Archive/Do Not Backup" mac01.systemsboy.com:/Volumes/Work/ /Volumes/Backups/mac01 --eahfs > ~/Log/mac01-backup-log.txt

Mirroring

The above uses of rsync, as I mentioned before, will not delete files from the target that have been deleted from the source. They will only propagate changes that have occurred on the existing files, but will leave deleted files alone. They are semi-non-destuctive in this way, and this is often useful and desirable. Eventually, though, rsync backups will begin to consume a great deal of space, and after a while you may begin to run out. My solution to this is to periodically mirror my sources and targets, which can be easily accomplished with the --delete option. This option will delete any file from the target not found on the source. It does this after all other syncing is complete, so it's fairly safe to use, but it will require enough drive space to do a full sync before it does its thing. Here's our network command from above, only this time using the --delete flag:

/usr/bin/rsync -az -vv -e SSH --exclude "Archive/Do Not Backup" mac01.systemsboy.com:/Volumes/Work//Volumes/Backups/mac01 --delete --eahfs > ~/Log/mac01-backup-log.txt

Typically, I run the straight rsync command every other day or so (though I could probably get away with running it daily). I create the mirror at the end of each month to clear space. I back up about a half dozen machines this way, all from two simple shell scripts (daily and weekly) called by cron.

Conclusion

I realize that this is not a perfect backup solution. But it's pretty good for our needs, given what we can afford. And so far it hasn't failed me yet in four years. That's not a bad track record. Ideally, we'd have more drives and we'd stagger backups in such a way that we always had at least a few days backup available for retrieval. We'd also probably have some sort of backup to a more archival medium, like tape, for more permanent or semi-permanent backups. We'd also probably keep a copy of all this in some offsite, fireproof lock box. I know, I know. But we don't. And we won't. And thank god, 'cause what a pain in the

ass that must be. It'd be a full time job all its own, and not a very fun one. What this solution does offer is a cheap, decent, short-term backup procedure for emergency recovery of catastrophic data loss. Hard drive fails? No trouble. We've got you covered.

Hopefully, though, this all becomes a thing of the past when Leopard's Time Machine debuts. Won't that be the shit?

1. According to the RsyncX documentation, you should not need to do this, because the RsyncX installer changes the command path to its custom location. But if you'll be running the command over the network or as root, you'll either have to change that command path for the root account and on every client, or network backups will fail. It's much easier to simply put the modified version in the default location on each machine.

2. Updates to Mac OS X will almost always overwrite this custom version of rsync. So it's important to remember to replace it whenever you update the system software.

Icons Icons Icons!

Not long ago a fabulous post on doodling appeared at one of the blogs I frequent on a regular basis, Subtraction. Khoi Vihn, the site's author, posted some absolutely lovely doodles he'd made, and I was instantly transported to my note-taking, doodle-drawing grad school days. It was kind of magical, and I really liked that idea for a blog post. And while I've decided I'm way too lazy and busy at the moment to scan my doodles, I did feel TASB could use a splash of the creative. I am, in addition to being a SysAdmin, a visual artist, after all.

Since this site is primarily about systems, it seemed appropriate to focus on an art form specifically related to the computer. And since I dabble in icon creation, and have worked up quite a collection over the years, I've decided to post a little sampler, and talk about my process. Just for a wee change of pace. Just for fun.

This is the first icon I ever made. I labored on this thing for days, trying to get that shiny, translucent, jelly bean look so prevalent on the Mac. Yes. It totally sucks major ass.

My First Icon: It Sucks Major Ass
(click image for larger view)

I quickly outgrew the glossy, candy-coated style — especially once I realized that I sucked at, and didn't enjoy making them — and started developing my own. I have two basic styles. The first one to emerge was a very flat, geometric style. My process is not very elegant, but I've used it for some time now and it works so I stick with it. Basically all the drawing is done in Illustrator. When I get a version I like I simply copy and place it into a 533x533 pixel Photoshop document. Typically, some color correction is required after the transfer. Mainly, the blacks get washed out, so I have a Photoshop action that selects the black range and drops it down to zero. Then the image gets resized to 128x128 pixels. Finally I use IconFactory's IconBuilder 5.1 plug-in (no longer current) for Photoshop to export the image to the icon format. Typically, here I just do a "QuickBuild" (which, unfortunately seems to be missing from later versions of IconBuilder). I don't really worry about creating multiple versions of my icons for different icon sizes and views. I just try to make icons that scale reasonably well. This is for fun after all. I'm not a pro, nor do I aspire to be.

Geometric Icons: Not Quite as Sucky
(click image for larger view)

The second style, and the one I generally prefer to work in nowadays, is more hand-drawn and cartoonish. Still somewhat flat, with some shading but no gradients and far less geometry than the above. These are all done using my trusty WACOM tablet, which I simply adore. Working this way allows me much greater expressiveness through line weight and shape, and through the individuality of my own particular drawing style. Plus it's just way more fun, though it can be a lot more work to get everything looking just right.

Hand-Drawn Icons: My Personal Faves
(click image for larger view)

A lot of these icons were intended as drive icons. I have a few different computers I work on regularly, and I tend to file-share between them a lot. Each has a SysApps partition and Work partition. Labeling each SysApps and Work drive on each system, while being aesthetically pleasing, also has the added advantage of making it very easy to distinguish which SysApps or Work drive I'm accessing at a glance when file-sharing. It's both practical and pretty. (Well, I think so anyway.)

Marx Brothers Icons: I Got Paid!
(click image for larger view)

Recently someone took notice of some of the icons I'd used in he lab and commissioned me to make him a custom set based on the Marx Brothers, he being a big Marx brothers fan and all. We worked together to figure out what he would like, and he gave me all kinds of resources and suggestions for the project, but I had a lot of freedom. It was the first and only job I've ever had like that. The first time I've ever had to create something visual for someone other than myself, but he was very cool to work with, I liked the challenge, and it was a really fun process. Basically I worked on it in my spare time, and I'd show him progress sketches every so often. He'd give me feedback and I'd go work on them some more and show him the results when things got good. The thing I liked best about the project was this outside feedback. It really kept me from being lazy. And in the end the icons were far more polished than I think they ever would have been had I just been working on them alone. I'm kind if a loner in a lot of ways, both professionally and personally. Not a big collaborator. But every now and then it's great to have some outside eyes. Some second opinions.

Which, now that I think about it, may be the whole reason to have a blog.

Directory Access Via the Command Line

I recently finally had occasion to learn some incredibly handy new command-line tricks I've been wanting to figure out for some time. Namely, controlling Directory Access parameters. I've long hoped for and wondered if there was a way to do this, and some of my more ingenious readers finally confirmed that there was, in the comments to a recent article. And now, with initiative and time, I've figured it all out and want to post it here for both your and my benefit, and for the ages (or at least until Apple decides to change it).

The occasion for learning all this was a wee little problem I had with my Mac OS X clients. For some reason, which I've yet to determine, a batch of them became hopelessly unbound from the Open Directory master on our network.

Weird Client Problem: "Some" Accounts Available? Huh?

(click image for larger view)

The solution for this was to trash their DirectoryService preferences folder, and then to rebind them to the server. This was always something I'd done exclusively from the GUI, so doing it on numerous clients has always been a pain: log into the client machine, trash the prefs, navigate to and open the Directory Access application, authenticate to the DA app, enter the OD server name, authenticate for directory binding, and finally log back out. Lather, rinse, repeat per client. Blech! The command-line approach offers numerous advantages, the most obvious being that this can all be scripted and sent to multiple machines via Apple Remote Desktop. No login required, no GUI needed, and you can do every machine at once.

The command-line tools for doing all this are not exactly the most straightforward set of commands I've ever seen. But they exist, and they work, and they're quite flexible once you parse them out. The first basic thing you need to understand is that there are two tools for accomplishing the above: dscl and dsconfigldap. The dsconfigldap command is used to add an LDAP server configuration to Directory Access. The dscl command adds that server to the Authentication and Contacts lists in Directory Access, and is used to configure the options for service access.

So typically, your first step in binding a client to an OD master in Directory Access is to add it to the list of LDAPv3 servers. This can be done via the command-line with dsconfigldap, like so:

sudo dsconfigldap -s -a systemsboy.com -n "systemsboy"

We like to use directory binding in our configuration, and this can be accomplished too:

sudo dsconfigldap -u diradmin -i -s -f -a systemsboy.com -c systemsboy -n "systemsboy"

The above command requires a directory administrator username and interactively requests a password for said user. But if you want to use ARD for all of this, you'll need to supply the password in the command itself:

sudo dsconfigldap -u diradmin -p 'DirectoryAdmin_Password' -s -f -a systemsboy.com -c systemsboy -n "systemsboy"

Directory Access: Adding an OD Server Configuration

(click image for larger view)

So, there you have it. You've now added your OD master to your list of LDAPv3 servers. You can see this reflected immediately in the Directory Access application. But, unlike in DA, the command does not automatically populate the Authentication and Contacts fields. Your client will not authenticate to the OD master until you have added the OD server as an authentication source. To do this you use dscl. You'll need a custom Search Path for this to work. You may already have one, but if you don't you can add one first:

dscl -q localhost -create /Search SearchPolicy dsAttrTypeStandard:CSPSearchPath

And now add the OD master to the Authentication search path you just created:

sudo dscl -q localhost -merge /Search CSPSearchPath /LDAPv3/systemsboy.com

Directory Access: Adding an OD Authentication Source

(click image for larger view)

If you want your OD server as a Contacts source as well, run:

sudo dscl -q localhost -merge /Contact CSPSearchPath /LDAPv3/systemsboy.com

Again, this change will be reflected immediately in the DA application. You may now want to restart Directory Services to make sure the changes get picked up, like so:

sudo killall DirectoryService

And that's really all there is to it. You should now be able to log on as a network user. To test, simply id a know network-only user:

id spaz

/>If you get this error:

id: spaz: no such user

Something's wrong. Try again.

If all is well, though, you'll get the user information for that user:

uid=503(spaz) gid=503(spaz) groups=503(spaz)

You should be good to go.

And, if you want to view all this via the command-line as well, here are some commands to get you started.

To list the servers in the configuration:

dscl localhost -list /LDAPv3

To list Authentication sources:

dscl -q localhost -read /Search

To list Contacts sources:

dscl -q localhost -read /Contact

A few things before I wind up. First, some notes on the syntax of these commands. For a full list of options, you should most definitely turn to the man pages for any of these commands. But I wanted to briefly talk about the basic syntax, because to my eye it's a bit confusing. Let's pick apart this command, which adds the OD master to the configuration with directory binding and a supplied directory admin username and password:

sudo dsconfigldap -u diradmin -p 'DirectoryAdmin_Password' -s -f -a systemsboy.com -c systemsboy -n "systemsboy"

The command is being run as root (sudo) and is called dsconfigldap. The -u option tells the command that we'll be supplying the name of the directory admin to be used for binding to the OD master (required for such binding). Next we supply that name, in this case diradmin. The -p option allows you to specify the password for that user, which you do next in single quotes. The -s option will set up secure authentication between server and client, which is the default in DA. The -f option turns on ("forces") directory binding. The -a option specifies that you are adding the server (as opposed to removing it). The next entry is the name of the OD server (you can use the Fully Qualified Domain Name or the IP address here, but I prefer FQDN). The -c option specifies the computer ID or name to be used for directory binding to the server, and this will add the computer to the server's Computers list. And finally, the -n option allows you to specify the configuration name in the list of servers in DA.

Now let's look at this particular use of dscl:

sudo dscl -q localhost -merge /Search CSPSearchPath /LDAPv3/systemsboy.com

Again, dscl is the command and it's being run as root. The -q option runs the command in quiet mode, with no interactive prompt. (The dscl command can also be run interactively.) The localhost field specifies the client machine to run the command on, in this case, the machine I'm on right now. The -merge flag tells dscl that we want to add this data without affecting any of the other entries in the path. The /Search string specifies the path to the Directory Service datasource to operate on, in this case the "Search" path, and the CSPSearchPath is our custom search path key to which we want to add our OD server, which is named in the last string in the command.

Whew! It's a lot, I know. But the beauty is that dscl and dsconfigldap are extremely flexible and powerful tools that allow you to manipulate every parameter in the Directory Access application. Wonderful!

Next, to be thorough, I thought I'd provide the commands to reverse all this — to remove the OD master from DA entirely. So, working backwards, to remove the server from the list of Authentication sources, run:

sudo dscl -q localhost -delete /Search CSPSearchPath /LDAPv3/systemsboy.com

To remove it from the from the Contacts source list:

sudo dscl -q localhost -delete /Contact CSPSearchPath /LDAPv3/systemsboy.com

And to remove a directory-bound configuration non-interactively (i.e. supplying the directory admin name and password):

sudo dsconfigldap -u diradmin -p 'DirectoryAdmin_Password' -s -f -r systemsboy.com -c systemsboy -n "systemsboy"

If that's your only server, you should be back to spec. Just to be safe, restart DirectoryService again:

sudo killall DirectoryService

If you have a bunch of servers in your Directory Access list, you could script a method for removing them all with the above commands, but it's probably easier to just trash the DirectoryService prefs (in /Library/Preferences) and restart DirectoryService.

Lastly, I'd like to end this article with thanks. Learning all this was kind of tricky for me, and I had a lot of help from a few sources. Faithful readers MatX and Nigel (of mind the expla

natory gap fame) both pointed out the availability of all this command-line goodness. And nigel got me started down the road to understanding it all. Most of the information in this article was also directly gleaned from another site hosted in my home state of Ohio, on a page written by a Jeff McCune. With the exception of a minor tweak here and there (particularly when adding Contacts sources), Jeff's instructions were my key to truly understanding all this, and I must thank him profusely. He made the learning curve on all this tolerable.

So thanks guys! It's help like this that makes having this site so damn useful sometimes, and it's much appreciated.

And now I'm off to go bind some clients command-line style!

UPDATE:

Got to test all this out real-world style today. Our server got hung up again, and we had the same problem I described at the head of this article. No one could log in. So I started trying to use the command-line to reset the machines. I had one major snag that caused it all to fail until I figured out what was going on. Seems I could not bind my machines to the server using the -s flag (secure binding). I had thought that this was the default, and that I was using it before, but now I'm not so sure. In any case, if you're having trouble binding or unbinding clients to a server, try the dsconfigldap command without the -s flag if you can, like so:

sudo dsconfigldap -u diradmin -p 'DirectoryAdmin_Password' -f -a systemsboy.com -c systemsboy -n "systemsboy"

That's what worked for me. I'm a little concerned that this is indicative of a problem on my server, but now's not really the time to go screwing with stuff, so I'll leave it alone for the time being.

This update brought to you by the little letter -s.

Adobe Legal Hosed My System

So recently there's been a bug in, according to Adobe anyway, Mac OS X that causes certain files in certain Adobe CS2 applications to wreak untold havoc on HFS+ filesystems. In a fairly recent Adobe Support Knowledgebase article on this issue Adobe says, and I quote:

Background information

Mac OS X causes illegal file names to be reported when it reads some of the font data used in the Vietnamese End User License Agreements, which are installed in the Legal or Legal.localized folders. This problem causes severe file system and hard disk corruption if the files are not deleted or if the file system is not repaired.

Apple fixed this problem in Mac OS X 10.4.7.

The fix for this, Adobe goes on to say, is to get rid of any folder in any Adobe CS2 application folder called "Legal" (in the Finder) or "Legal.localized" (in the Terminal), and then run Disk Utility to repair the disk. They also suggest that upgrading to Tiger 10.4.7 or later is a good last step as it halts the corruption process.

I'd actually had the problem last summer on my lab systems, which had exhibited evidence of it during a clone operation. Any clone of a system would fail, and the asr command would actually report the name of the trouble file. Indeed, running Disk Utility's "Repair Disk" (or in our case, fsck from single-user mode) would fix the problem and our clones would subsequently succeed. Those systems were running 10.4.7.

My office system, on the other hand, never went through the cloning process, so I never detected the problem. But I seem to have been bitten by this bug and in a bad way. Please note the last sentence in the above quote:

This problem causes severe file system and hard disk corruption if the files are not deleted or if the file system is not repaired.

Yesterday I was running Disk Utility on a problem drive and decided to run it on my system partition as well, for good measure. This was the output:

Disk Utility: Reports Unfixable "Illegal name"
(click image for larger view)

See that "Illegal name" error in bright red? That's a telltale sign that you've got the "Adobe Legal bug" (as I like to call it). It's also, I can tell you from cold, hard, agonizing experience, a telltale sign that you are indeed fucked. Hard. I think this is what Adobe is referring to as "severe file system and hard disk corruption." I tried everything to make that "Illegal name" error go away. Actually, attempts to fix the problem took more time than it took for me to rebuild my system, which is what I ultimately had to do. Now I hate rebuilding systems. Almost as much as poking myself in the eye with white hot forks. So I spent the better part of the day attempting to fix the problem.

The first thing I did was to boot into a good system partition. I just happened to have a base Tiger OS installed on a firewire drive, so I booted into that. I then went through the aforementioned Disk Utility and fsck routines. (I also tried Disk Utility from the Tiger DVD, just to be thorough.) No luck. I always got the "Illegal name" error. I also tried fsck with some options to see if I could actually track down the file with the illegal name and delete it. Running:

sudo fsck_hfs -frd /dev/disk0s10

Produced the following output:

** /dev/rdisk0s10** Checking HFS Plus volume.** Checking Extents Overflow file.** Checking Catalog file.** Rebuilding Catalog B-tree.hfs_UNswap_BTNode: invalid node height (1)** Rechecking volume.** Checking HFS Plus volume.** Checking Extents Overflow file.** Checking Catalog file.Illegal nameillegal name is 0x00 54 00 69 00 65 03 02 03 01 00 6E 00 67 00 20 00 56 00 69 00 65 03 02 03 23 00 74 00 2E 00 68 00 74 00 6D 00 6Creplacement name is 0x00 54 00 69 00 65 03 02 03 01 00 6E 00 67 00 20 00 56 00 69 00 65 03 23 03 02 00 74 00 2E 00 68 00 74 00 6D 00 6C** Checking multi-linked files.** Checking Catalog hierarchy.** Checking Extended Attributes file.** Checking volume bitmap.** Checking volume information.Verify Status: VIStat = 0x0000, ABTStat = 0x0000 EBTStat = 0x0000        CBTStat = 0x0000 CatStat = 0x8000** Repairing volume.replacement name already existsduplicate name is 0x00 54 00 69 00 65 03 02 03 01 00 6E 00 67 00 20 00 56 00 69 00 65 03 23 03 02 00 74 00 2E 00 68 00 74 00 6D 00 6CFixIllegalNames - repair failed for type 0x23B 571** The volume SysApps could not be repaired.volume type is pure HFS+primary MDB is at block 0 0x00alternate MDB is at block 0 0x00primary VHB is at block 2 0x02alternate VHB is at block 66846718 0x3fbfffesector size = 512 0x200VolumeObject flags = 0x07total sectors for volume = 66846720 0x3fc0000total sectors for embedded volume = 0 0x00

This seemed to suggest that, yes, there is an illegally named file somewhere, but that it can't be replaced because the replacement name already exists. Ew! I'm not sure what that means, but it does not sound good.

Undaunted (alright, maybe a little daunted), I decided to try cloning the system to see if I could get the name of the illegal file like I did last summer, using the asr command. I also thought it was possible that any filesystem damage, depending on the nature of that damage, might be repaired by cloning the system to a clean, undamaged filesystem. So I created a 40GB disk image, which actually took quite some time, probably because I was booted from a slow firewire drive. But it finally completed, and once it did I cloned my sick system partition to it. This also took a great deal of time over firewire. Like hours, actually. Like I actually had a good excuse to go to lunch for once. But it did finish successfully, and it never reported an illegal name in the process of cloning. So I ran Disk Utility on it, hoping that maybe the new filesystem did the trick. No such luck. Same error.

By this time I'd spent — wasted, actually — an entire day on this problem. A problem apparently caused by "font data" robbed me of an entire day of productive work and put me in a nasty mood to boot. My options spent, I did the thing I hate to do so very much: I rebuilt.

There is at least one silver lining to all this, and that is the magic of disk partitions. You see, it's for reasons just such as these that I partition my user data from my system and application data. Every Mac system I build has a partition called SysApps — that houses the system components and applications — and one called Work — that houses all the data I generate, from home account preferences to Final Cut files. In the above scenari o it was the SysApps partition that was corrupted, but the Work partition checked out just fine. This two-partition method offers numerous advantages in such a scenario. For one, after booting into the firewire drive it was a simple matter of telling NetInfo Manager to use my home account on the Work partition and I was right back to work checking email and the like, which is pretty crucial for me to be able to do. All my setups and preferences worked normally, my browser bookmarks were all there, my calendars were all intact, and I could at least work on actual work instead of being stopped dead in my tracks by this corruption problem. Secondly, in this case reformatting the SysApps partition was necessary. Had all my data been on that partition I would have had to back it all up somehow. And what if that corruption lay somewhere among my user data? I'm not sure what I would have done then. I may just have been screwed. But because my data was walled off, it was a non-issue. Thirdly, my Work data takes up the majority of the data on my system, and it's quite large — about 170GB. In a single-partition system I'd have had to blow all that away and restore it somehow. Backing up and restoring 170GBs of data takes a long time and would have significantly increased the amount of time I spent getting back on my feet. With my two-partition system, about 30GB was all I had to worry about. And all that cloning I did in the hopes of finding a fix? With my 30GB SysApps partition it was painful and time-consuming, but it was doable (though whether it was worthwhile is debatable). If I'd had to do that with 200GB of system, application and user data combined it would have been downright impossible.

Restoring the SysApps partition was a pain, to be sure. But there was nothing there that didn't already exist somewhere among the myriad software installation discs in my office, so it wasn't so bad. And there was a lot of stuff I could restore right from the backup I'd made with asr — things like drag-n-drop applications, crontab and host files, and the like. Troubleshooting the problem took about a day, but rebuilding the system took a few short hours, in part because most of my preferences reside in my perfectly preserved home account. It was mostly just a matter of cloning a working system (from my base Tiger install on my firewire drive), reinstalling a few broken applications (Final Cut, Adobe stuff, etc.), and running Software Update as needed. Along the way I checked the health of my SysApps drive, and no "Illegal name" errors were reported. Phew!

So that's the saga of how Adobe Legal hosed my hard drive. I'm not sure if the blame really lies with Apple or Adobe. What I do know is that I'm sticking with my two-partition system, and I'm permanently deleting all those "Legal" folders associated with Adobe products. I suggest you follow this latter step yourselves, 'cause, if left untreated, it's true what they say: It really can cause severe filesystem and hard disk corruption. I'm living proof.