Tiger Lab Migration Part 8: Home Account Woes

Boy have we got troubles. Sure I tested everything. Sure I did everything I could to make sure this upgrade went smoothly. But does that ever matter? No. It doesn't.

Everything tested out fine. Tiger was able, from day one, to mount our NFS home account server -- a network RAID known as the Panasas. It was able to read files, to write files. It seemed fine. But we'd had our fair share of troubles with the Panasas, even in the beginning, even in Panther. Certain Macromedia products -- actually, Flash MX 2004, to be specific -- had certain functional disabilities. In particular, when any user was logged onto the Mac as a network user whose home account was located on the Panasas, he would find himself unable to publish HTML previews from within the Flash MX 2004 application. The error message was something along the lines that Flash was unable to read the HTML templates, which live in the user's home account. Oddly, on first launch, Flash could write the templates, but for some reason, it could not read them for the preview. If you have any Flash knowledge whatsoever (which I don't -- I was informed by student experts) this renders Flash essentially useless. The problem did not, however, exhibit itself with network home accounts mounted on Apple shares, neither via AFP nor NFS. So the workaround was to create a special account for Flash developers, which they could log into when doing Flash work, and which was mounted directly on the MacServer and shared via AFP. Hence was born our FlashDev account, which has worked fine all year.

Jump ahead to the present: we've upgraded to Tiger. It seems to have gone fine. Suddenly, certain apps begin acting strangely. Illustrator can't save files. Word also has trouble saving files, and behaves erratically, complaining of "permissions problems" and the inability to write temp files to a "network disk." Final Cut, too, begins acting flakey. Shit. It's like one of these sci-fi-horror flicks where the brain transfer experiment seem to have gone perfectly, but then, gradually, the patient begins acting... Different somehow...

And then all Hell breaks loose.

Tracking down all the problems, I finally came to the conclusion that a number of applications are unable to read their preference files in Tiger when those files reside on the Panasas. This perhaps causes a chain reaction that makes these apps unable to properly write or save files: FCP writes 0 byte files; Illustrator CS just can't save your document. Oddly, Illustrator CS2 doesn't seem to suffer from this problem, and all these apps (except, of course, Flash) seem to work properly from within the Panther environment.

Seems to me like a nasty reaction between Tiger and the Panasas home account server.

I've done some testing, and I plan to do more. So far, I've tried booting into a Panther client and found that the problem is greatly reduced in Panther (though it always existed to some extent, particularly, again, with Flash) and exacerbated in Tiger. I've also tried resharing the Panasas NFS export via AFP from the MacServer. I was fairly sure this would provide a reasonable, if temporary, workaround, but it didn't. I want to try it again, just to be sure, but if resharing via AFP does not provide a cure, it would indicate that the fault lies somewhere with the Panasas. The other possible culprit is Tiger's implementation of NFS. To test this, I am building a more generic (non-Apple, non-Panasas, non-proprietary) NFS server to test from, on a Linux install on my new Dell laptop.

So, best laid plans, and all that. What a frickin' mess.

I'll keep you posted as I learn more.

UPDATE 1:
I built a Linux install for the express purpose of testing networked Mac home accounts on a non-proprietary (non-Mac, non-Panasas) NFS mount. When running a Mac home account from this mount, the Final Cut problem disappeared, and the application acted normally. The same was true for Microsoft Word. Illustrator CS still behaved badly, though, as did Flash MX 2004. It's clear now that these problems result both from Tiger's implementation of NFS as well as Panasas's. So, the breakdown goes something like this:

• The Flash problem is probably Flash/NFS-specific (though possibly Mac-specific as well)
• The Illustrator problem is Tiger/NFS-specific
• The Final Cut and Word problems are Tiger/Panasas-specific

None of this makes my job any easier. It kind of sucks to realize that there is no magic bullet, because that means that there probably is no single fix. Still, it's useful to know.

UPDATE 2:
The issue with Final Cut is not a matter of the application being unable to read files; it's a matter of it being unable to write them. Essentially, FCP cannot write a preference file. Launching with no preference file, FCP creates a 0 byte preference file in the user's home account. Also, if you try to save the FCP project file to the NFS share, it will be a 0 byte file and unopenable. Saving the FCP project file to a local volume results in a perfectly usable FCP project file. The complete inability to write files seems to be unique to Final Cut. Other applications (Illustrator, Flash) seem to have trouble reading files, but writing them seems to work fine (which is why I assumed this was the case in FCP). Word (now updated to 11.2) also has trouble writing files, but only when saving an open, modified document. That is, if you create a new document in Word, and hit "Save," it saves just fine. If you keep the document open, make changes, then hit "Save" again, it complains that it can't write to the shared disk. Hit "Save" again, and Word will create a new document and use the first string of characters in the document as the name. "Save As..." however, works as expected.

UPDATE 3:
It occurs to me (through the suggestion of other, smarter folks) that maybe this is not completely an NFS problem, particularly in the case of the apps that have problems writing files (Word and FCP). Those apps work just fine when we mount our homes on a generic NFS export. So it's possible that the write problems we're having are filesystem-related, not NFS-related. This theory holds a lot of water considering the fact that the Panasas uses its own, proprietary file system.

UPDATE 4:
I've been in contact with the Panasas folks, and I'm working with them in trying to determine the problem. So far, I've sent them tcpdump results from both the Panasas and the Mac while FCP tries to save a file. I'm told that there are a "massive number of 'file not found' type errors" from the dump. That can't be good. I've also sent them a few file listings from the home account of the user in question. Meanwhile, I'm wondering if the next Tiger update will do anything to correct these issues, or if upgrading the Panasas shelf would help, while at the same time, trying to come up with a reasonable workaround, in case Panasas is unable to reslove the issue. The only thing I can think of is to move the Mac home accounts to another server. Which would suck hard. Two steps backward.

UPDATE 5:
Still in constant contact with Panasas. I'm sending them log files now, and planning to do a ktrace, which will be a new experience for me. Also, I'm seeing this issue occur with other applications now. In particular, Macromedia Director MX 2004 is unable to write files to the Panasas mount. This is particularly odd because under Panther that application had a completely different problem: it just wouldn't launch. So I'm currently dealing with three seperate incident reports: one for the problem noted here, one for a blade crash that happened a couple of weeks ago, and one for the original Flash problem from our Panther days. It's a pain. Fortunately, the Panasas people are very nice, professio nal, and helpful.

UPDATE 6:
So, this does seem to be a filesystem issue at its core. Sort of. After much work with the Panasas guy, we've narrowed down the problem considerably.

Mac applications write temporary files -- files that are in use but that haven't yet been saved -- to various locations depending on the app. Final Cut writes its temporary files to a folder called .TemporaryItems at the top of whatever volume the user is working from. So in our model, the top volume is /home, and the sboy home account is inside /home. The whole thing looks like this: /home/sboy. Final Cut wants to write its temp files in /home/.TemporaryItems. And it is able to do this. If I look in /home/.TemporaryItems, I find all the files I've been unable to save to my user account, perfectly intact. So, temporary files go in /home/.TemporaryItems, and saved files in /home/sboy. Now, here's the grind: The Panasas setup requires that each home account be a seperate volume. So, /home is one big volume, and /sboy is another volume inside /home. This means that when Final Cut tries to move the temporary files from the .TemporaryItems folder, which is located on the volume /home, to the sboy user account, which is another volume at /home/sboy, it is moving the files across filesystems. And we get an error.

Interestingly, moving files across filesystems when Panasas is mounted on a Linux box produces the same error, but Linux does something to compensate for this problem, and is then able to save the file. But the error still gets produced. It's just that Mac OS X 10.4.2 (this did not happen in Panther) does not have the mechanism to correct for this error. It gives up and leaves the file in its original location: the /home/.TemporaryItems directory.

Also, this explains why everything works fine when I mount the user's home account on my Linux NFS export. In that case, everything is on the same volume, and there is no moving of files across filesystems. It's got nothing to do with NFS at all.

I'm curious to know why this doesn't happen in Panther. Is it possible that Panther has the same error correction function we see on Linux? What about earlier versions of 10.4? Do they have this mechanism? Will 10.4.3 have it? I surely hope so. I'll be looking into who I should contact at Apple about all of this.

Incidentally, the Panasas guys have figured all this out by examining ktrace and tcpdump files made while the program ran and the error occured. I did not figure any of this out on my own. All I did was run the commands and upload the files. Still, it's a nifty bit of sleuthing. I'm learing a lot.

UPDATE 7:
So, after mulling this over for a while, a rather obvious workaround occured to me. Currently, we mount the entire Panasas volume on our systems. This volume includes all home accounts within it, as well as the .TemporaryItems folder at its top level. Each user's home account on Panasas is a separate volume, and so is the top-level volume that contains .TemporaryItems, so saving files must cross filesystems (from the .TemporaryItems folder on /home to the user's folder). The workaround is to mount each user's home account individually. Doing so causes FCP to treat the user's home account as the active volume, and to use the .TemporaryItems folder in said home account. In this scenario, no filesystem boundaries are crossed, and no error occurs. I tried it, and it works. FCP (and Word, incidentally) can now both save files to the user's home account normally, and application preference stick. Final Cut is again usable.

Hallelujah!

I'm testing this on a couple machines before I go and announce it to the community and put it on all the Macs, just to make sure it works properly. But it's looking good. I'm glad to know that the past two weeks I've spent on this haven't been in vain.

The Dark Side Part 4: System Restore? What a great idea!

I'm trying to be as positive about my Windows experience as possible. It's not easy. But as the popups and virus alerts have subsided, I've managed to poke around and find some really useful stuff on my Windows system. Stuff that doesn't exist on the Mac. One of the most impressive of these is the System Restore feature. I haven't actually tested it yet, but from the looks of it, the way it works is, you open the System Restore application, and there are two options: either restore your system to the state it was in at an earlier date, or create a restore point, which is essentially a record of the state of your system at this particular moment in time. System Restore actually creates restore points on its own, at regularly scheduled intervals as well. On the fly. In the background.

Now I don't know how well this works, but it's a great idea. I would absolutely love something like this for Mac. On Mac, if you want to "restore" your system, you're essentially relegated to the "Archive and Install" procedure, which, you know if you've ever done one, sucks. Hard. Wouldn't it be great if we Mac users could apply the latest OS update, for instance, and, two days later, when we realized some key functionality is broken, simply run a System Restore-like application and be back to where we were in a few short minutes? I think it would. And I'm sure you agree.

I really hope Apple has something in the pipe like this. Since my first broken Mac OS X update, I've longed for a tool like System Restore, and it's been one of the few things I've actually envied the Windows folk for having.

My Hand to God: Adobe Space Monkey

I was attempting a drag-and-drop install of Adobe Photoshop CS2 to a machine that had not been licensed yet, here in the lab, just to see if one tiny part of my life could be made that much easier. It could not. The attempt, however, did afford me the opportunity to get a load of this Easter Egg:

Adobe Space Monkey Splash Screen
(click for larger view)

Which was promptly followed by this error message:

Adobe Space Monkey Error Message
(click for larger view)

Seems the Photoshop programmers have left us something to smile about when we discover that we'll be installing Photoshop on each individual Mac in our lab. Thanks, guys! You couldn't come up with a volume license scheme that allows me to install a single copy of PS across multiple machines, or a package installer that would let me do same over the network, but you managed to find the time and resources for Adobe Space Monkey. Awesome! I finally get why Photoshop costs hundreds of dollars: You're a bunch of assholes.

Thanks for clearing that up.

Yowsa! Browsers!

Yes, I know, that is the worst post title ever. Sorry.

Let's move on.

I've been playing with the beta version of Firefox. I must say, it's pretty cool. Lots of nice stuff. A few bad things as well. I've compiled a list of the systemsboy-specific pros and cons. In doing so, I've gone over what it is that makes me like certain browsers over others, and started looking at some additional options. Here is a loose, and very subjective, collection of ramblings and lists about browsers.

General
Browsers are a mixed bag. None seems to do everything you want it to do exactly how you like it. So I tend to use multiple browsers. For general surfing. I use Safari. It's the most comfortable for me to use, and the easiest on the peepers. Firefox, however, is the most full-featured. It will load just about anything properly. I use it for editing the blog, as well as whenever a site doesn't appear properly in Safari, but it's kind of slow and clunky looking. At this moment, I am using Camino, which I'm surprised to discover, actually supports Blogger's "Compose" mode. It's pretty, and I like the key commands. And, hoo-boy, it's fast. Camino, however, is one of those eternally-beta-level (okay, technically it's "alpha," but whatever) browsers, and I'm pretty sure it won't be able to totally replace Safari or Firefox for me. Even writing this post, I had a minor glitch. But I'd love to see Camino development continue to the point where the browser is as rock-solid and feature rich as Firefox, and as pretty as Safari. That would be great.

When trying out browsers, I tend to consider a few basic parameters that determine how much I like or dislike the browser in question. These parameters vary among users and are extremely subjective. The ones that affect me the most are:

• Functionality -- the ability to load pages, web apps, forms, etc.
• Appearance -- how the UI looks, as well as how pages render
• Usability -- how well key-commands and UI elements, like preference panes, function and make the browser comfortable and easy to use

Now I'd like to take a quick look at the browsers themselves, and list some of my likes and dislikes for each.

Safari
Despite everything, this is my favorite all around browser. I use it for everyday surfing and most web-related tasks. Still, there are things I've grown to hate about Safari.
The Good:

• Safari is pretty -- the UI is nice, and pages, when they load properly, look great
• Safari is fairly fast, at least fast enough for me
• Safari's key commands, for whatever completely subjective reasons, make the most sense to me
• Safari is bundled with the OS, so I can use it on any Mac I may be on (lab, client, etc.)

The Bad:

• A recent Safari bug is the window placement problem I've blogged about that drives me nuts, where Safari resets my window placement after quitting and relaunching the app
• Another recent Safari problem has cropped up wherein pages don't load the first time, particularly when lodaing a group of tabs
• Safari still lacks the ability to bookmark a group of tabs, a feature I've been waiting forever and a day for
• This frickin' scroll-wheel jump is frickin' annoying
• Blogger's "Compose" mode is not available in Safari

Firefox 1.0.6
Firefox is the most full-featured and configurable of the Mac browsers. I like it a lot, and I use it to post to this site as it works with Blogger's "Compose" mode. It features a "Multiple Home Pages" function that I could see coming in real handy someday. It's very stable, and works almost identically on Mac, Windows and Linux.
The Good:

• Works across platforms
• Can load just about anything
• Stable and secure
• Just about anything you can dream of is configurable (except, unfortunately, key commands)
• Firefox is skinnable, which is great if you can find a theme you like, which is hard (I'm using "Brushed" right now, which I like, but not as much as "Pinstripe," which appears to no longer work with Firefox)
• Firefox's key-commands aren't bad once you get used to them, which doesn't take long
• Firefox can use Amazon's A9 online bookmark manager, which helps mitigate the ongoing problem of inconsistent bookmarks across multiple computers

The Bad:

• Firefox is kind of ugly -- not butt ugly, but clunky and silly looking, comparatively speaking
• Firefox is slow, both at launch and loading pages
• Clicking in the URL field selects the entire link, instead of placing the cursor at the click-point
• Firefox has no key-command to stop page loading (that I can find, anyway)
• Opening a URL in a new tab requires a command- or control-click and can't be done from the middle mouse button
• The enormous list of preferences is badly displayed and hard to manage from the clunky sheet interface provided
• The key-command for said preferences window usually doesn't work, requiring a trip to the application menu

Firefox 1.5 Beta
The new beta version of Firefox shows real promise, and addresses a lot of my issues with the previous version. It's the impetus for this post, actually. The reason I've started looking at browsers again.
The Good:

• Clicking the URL bar no longer highlights the entire field, but rather, puts the cursor right where you just clicked
• Clicking a link with the middle mouse button now opens the link in a new tab
• Tabs can be rearranged, which is just so cool
• The new, tab-based, floating preferences palatte is nicely done and easy to navigate
• The key-command for the preferences window now works consistently
• There is now a key-command to stop loading a page (and it's the standard "command-period")
• In fact, most key-commands now match those found in Safari

The Bad:

• The tab selection key-command (which used to be "control-tab") has reverted to the one used for Mozilla (which is "control-page up/down", which sucks for people, like me, who switch between a PowerBook and a Desktop, as the PowerBook version of the command requires the use of the "fn" key, and, thus is a differnt key-command than it is on a desktop machine), which was the reason I stopped using Mozilla in the first place -- this is a deal breaker for me
• This new version placed aliases of Firefox in my sidebar and on my Desktop without even asking -- like on Windows!

Camino 1.01a Alpha
Camino has gotten steadily better in recent months and years. It looks good, it feels good, and it's snappy as Hell. If it ever becomes stable enough to get out of alpha, I think it could be a contender. Right now, it is not what I'd call dependable. In fact it crashed in the middle of this post.
The Good:

• Camino is fast -- it feels the fastest of the browsers I've tested, though this is not a scientific assesment by any means
• Camino is pretty -- this latest version uses the "Unified" theme, which I personally think looks swell, and is aqua throughout
• Very configurable -- more than Safari, though not as much as Firefox
• Camino has good, easy-to-learn key-commands

The Bad:

• Can you say, "Crashes a lot?" That's my major beef with Camino
• Also, as it's perpetually in "alpha", there are probably lots of pages that don't load properly in Camino, but I haven'tused it as thoroughly as the others, so this is really only a cursory examination

So that's my take on the state of browsers for the Mac. I use a range of them for various tasks, which kinda sucks for the obvious reason that my bookmarks are in a state of total disarray, a problem only partially mitigated by the availability of services like Amazon's A9, or del.icio.us. Still, between Safari and Firefox I can get my work done without too much trouble. And I'm learning to let the bookmark thing go. Too many computers, too little time, and with the greatness that is Google, bookmarks are less crucial than they once were. I have to say, though, I'd love it if Google made an online bookmark manager. Or, Hell, a cross-platform browser with integrated, online bookmark manager. C'mon. You've got to admit, that would be sweet!

UPDATE 1:
Looks like there is a stable release of Camino after all. My bad. I've just been testing it. It has weird, centered, aqua tabs, and it's not nearly as fast as the alpha version. The alpha version is so fast, I get vaguely nauseous surfing with it. Dude. It's fast. But I found one other beef with Camino, and all the browsers I looked at: none of them renders text as nicely as Safari. Don't know why. All the non-Safari browsers render text slightly bolder. It makes everything look, I don't know, chunky and a little cheap, particularly on pages like MacFixit. I think this has a lot to do with my reasoning for using Safari for day-to-day surfing. It looks very nice. But if you need the speed, check out Camino 1.0a.

Google Blog Search and Blogger

Oh great. Blogger, apparently, has switched their search engine to the newly released Google Blog Search. And while this service does look immensely cool, and I have little doubt the Google folks will make it so, right now it doesn't work right. From what I can tell, Google Blog Serach, which is currently in beta, only goes back so far in time, and not quite far enough that all my posts are searchable. It's a cool engine, and it allows me to search by blog URL and article title, but if I search, for instance, "Tiger Beefs" in the blog title at the URL for this blog, I get nothing. Yet I'm sure I have an article called just that.

So, fine. I understand that Google Blog Serach is in beta, and may not be doing everything it should be doing at this point. Is there some reason, then, that the Blogger folks decided to make our formerly very functional search engine run on the currently beta, non-fully-functional Google Blog Serach?

I don't know. All I know is, I hope they fix this soon.

UPDATE 1:
About Google Blog Search acknowledges this problem:
"Why aren't my oldest posts listed?

Since Blog Search indexes blogs by their site feeds, it will only include items that have been posted since it started indexing a given blog. For most blogs, that will be around June 2005, or the time at which you submitted your blog for inclusion. We are working on ways to include older posts as well."

This really sucks for me, since my site goes back to May, and because one of my most popular posts is from then. So now, people relying on Google to find anything on this blog from May are screwed, and my site traffic is likely to go down. (So if your looking for my "Disable Spotlight" script, it's here.)

UPDATE 2:
Using regular old Google's advanced search yields far more accurate, if less pretty, results when searching the blog URL by term.

UPDATE 3:
Jeff Harrell, of The Shape of Days, is kind of a Nazi, but I read his site anyway, for the great writing and the occasional interesting post on technology. Recently he had a great writeup about Google Blog Search and the whole idea of tagging blog entries. He's got me thinking again about blog categories. I use categories mainly to organize my site into general chunks that are probably mainly useful mostly to me, so I'll probably keep using them in the sort of general way I do. But his thesis is correct: Tags or categories are kind of irrelevant to anything text-based, as text already contains the information you'd add as a tag, and is so easily searchable. For instance, searching my blog for the term "tag" would, in theory, yield this article, and any other article containing the term. So tagging this article with a "tag" tag would be pointless, because the information is already there.

Now all we need is a good search engine.