launchd and Lingon

Well, it's about time I wrote a post about launchd. launchd is the new catch-all services launcher in Tiger. It's the thing that makes everything you need in order to do anything on your computer launch, either at boot, or at the appropriate time. It's got some real problems and limitations right now, but hey, it's new. And there's plenty good to say about it, even in this early stage.

I won't go too much into how launchd works nor all of what it does. But if you want to get an idea of what services have been moved to launchd from other parts of the system, take a look at the files in:

/System/Library/LaunchDaemons

Here you'll find a list of preference files, or .plist files. These files are all named for services that used to be started and handled by init and other such UNIX daemons and config files. Among the most notable in this list are the periodic maintenance jobs once handled by cron. These, too, have now been taken over by launchd, though cron remains available for scheduling other jobs. For now.

Anyway, this should give you a good enough idea what launchd does. If you're really lost here, I suggest you do some homework. There is a great overview of launchd, complete with tutorial, over at AFP548. There's also a good launchd Developer document at Apple's Developer site. Between these two you should be able to get largely up to speed on launchd. And if all this services stuff is way over your head, well, just move along please.

One of the coolest features of launchd, though, is the fact that it is easily user-configurable, and user-specific. That is launchd items, or LaunchAgents and LaunchDaemons as they're called, can be placed in the user's home account and run on a per-user basis. The other remarkably cool feature of LaunchAgents and Daemons -- and the one I really want to talk about -- is that they can watch folders and/or files. These two features allow for a degree of conditional automation -- and by that I mean, If/when user does A, then System does B -- previously undreamt of on a Mac. Well, that may be an overstatement, but launchd will certainly make such customization a great deal easier. And it's not even that hard to use, once you get the hang of it.

If you check out the AFP548 demo, they have you create a LaunchDaemon that watches a folder for modification, and then, if the folder is modified, launchd automatically moves the items of this "watch folder" (and, yes, that is the technical term) to a new destination folder. Already you can see where this would be extremely useful for backups or versioning systems, or any kind of synchronization of files. The idea of the watch folder/file is remarkably powerful. You can have launchd run any shell script, AppleScript or command anytime any file or folder is changed. In our lab we were, at one point, using launchd to watch our list of automount user mounts, and automatically reload the automount daemon whenever a new user was added to the list. You could (and I may, at some point) make a launchd item that detects the mounting of external hard drives, and then runs a script to disable Spotlight on said drives. There are a million things you can do with this one feature alone. But to do any of it, you must know how to edit those pesky .plist files.

Being no stranger to editing XML .plist files, I started off the hard way, editing them by hand in a text editor. I then graduated to the Property List Editor included with the Developer Tools. But recently, I found an excellent and free utility called Lingon, which takes a lot of the grunt work out of creating LaunchDaemons and Agents. It even comes with an "Assistant" to help you get started creating your first LaunchDaemon. And it will blessedly load your items into launchd, so trips to the command-line and the arcane syntax of lines like this:

launchctl load -w /Volumes/Work/systemsboy/Library/LaunchAgents/com.systemsboy.LA01

become a thing of the past.


Lingon: Launch Your Dreams Right Here!
(click for larger view)

So, okay, sometimes a GUI is a great thing. No offense to the Lingon guys and/or gals, but I'd really love to see Apple include a front-end to launchd sometime in the future, so that it could be really accessible to mere mortals without all the fussing around. Until then, Lingon makes a great addition to any Admin's toolbox who might have need of the sort of auto-magic attainable with launchd.

I do want to note one more thing. It's possible to seriously lock up your system with LaunchDaemons and Agents. To wit, I somehow created a LaunchAgent that perpetually opened one of my hard drives. This, for all intents and purposes, prevented me from accessing any application but the Finder. In fact, I couldn't even navigate the Finder to remove the item as every time I tried to do anything at all, the window to the drive would open and take over the system. Nor could I easily access the offending LaunchAgent via single-user mode, as my home account is stored on a second partition that is not mounted in single-user mode. Fortunately I keep a secondary user account for just such emergencies. In the end, I was able to unload and delete the offending LaunchAgent via this secondary account. So just a word to the wise: Use launchd with care. And keep a secondary account handy, just in case.

And one last thing: Hats off to the Tiger devs. I bitch a lot about 10.4, but launchd is a keeper. One of the best hidden surprises to this new OS.

Tiger Lab Migration Part 10: Up and Running

Take it as a symbol of my incompetence. Take it as a sign of the inherent difficulties in unifying any highly cross-platform environment. Take it as an indication that major OS updates in said environments are often far more difficult than you might have ever guessed. Take it for what you will. I'm just happy and relieved to report that, after many months of difficulties, we seem to be out of the woods: Our Tiger Lab Migration is complete.

(Note the sound of me frantically knocking wood.)

There are several things that have transpired -- things mostly out of my control -- that have made this migration, at last, complete. If you recall, there were two initial major problems from which we were suffering at the outset of the migration: 1) Mac OS X 10.4.2 was unable to write files across filesystems, which was impeding our use of certain software, particularly Final Cut Pro, which is used heavily in our lab; and 2) our Panasas brand home account RAID (on which we host our networked Mac home accounts via NFS) was crashing, almost daily at the end there. Problem number one was mitigated by certain workarounds I was able to implement using mount_nfs and loginhooks rather than the preferred automount method for mounting home accounts on our client workstations. Problem number two had both myself and the Panasas engineers scratching our heads.

Ironically, both problems are solved by system updates.

The inability of Mac OS X 10.4 to successfully write files across filesystems is cured, thankfully, with the latest update to the OS: 10.4.3. I have not yet implemented 10.4.3 lab-wide as yet, and so we're still running my mount_nfs workaround on the client systems, but on my admin machine I am testing the new update, and so far so good. Things behave properly again. I will also be running the update and the automount method on a test system on the floor for good measure, but I'm reasonably confident that this issue can, at last, be put to rest.

The home account server crashes also seem to have been cured by updating the Panasas RAID OS software. We upgraded on Friday, November 4th, 2005, and have not had a single crash since. This could be mere coincidence, but I'd be pretty surprised if that were the case. The crashing seems to have stopped, and I'd bet a fair chunk of cash that the update is the reason why this is so.

The best part of all this is that, for the past week, the lab has been quiet. People come in. They log on to the systems. They do their work. And everything functions properly, without incident, for the first time in months. The knocks on my door, the panicked moments, the hair-pulling-filled long nights and weekends have all subsided. My stress levels are slowly returning to what they once were, which is not to say normal, but rather normal for me. It is truly a thing of beauty.

There are two things a SysAdmin likes best in the world: a good challenge, and successfully overcoming said challenge. The Tiger Lab Migration has afforded me a hefty dose of both. And despite all the trauma, it's been an enlightening and illuminating experience from which I've learned a very great deal.

File Creation Dates Hosed Copying from Mac to Windows

I just wanted to draw some attention to a potentially fairly nasty Finder/Mac OSX bug that hasn't gotten the attention it probably should. A few days ago I posted on the latest Tiger release, what it fixes, what it doesn't. And in the comments section of that article, two people wrote in about a problem wherein the Finder loses the creation dates of files saved to MS-DOS formatted disks. That is, any file saved to a Windows formatted disk (and I don't know if this only applies to fat-32 disks or all Windows formats, but I would suspect the latter) will lose its creation date, which will be set to nothing. It will simply be blank, from my understanding of the problem. And please be aware, I have not experienced this problem myself. But a number of people have. We don't hear about it much in the Mac community, because much of the Mac community tends to work only on Macs. To be affected by this bug, you'd have to be both working from Mac to Windows, and have a serious need for file creation dates. This is not everyone, but it is a serious problem nonetheless, and Apple should hear about it. Though I have not personally suffered from this problem (yet), I sincerely sympathize with folks who need to get things to work cross-platform. I have the need myself, and it can be a real bitch. But when basic functionality like the preservation of file creation dates begins to fail when working cross-platform, it's way more than a bitch. It's a fucking nightmare. A lot of people turn to Apple for cross-platform compatibility, because -- let's face it -- no one else is doing half as good a job as Apple in this arena. But sometimes it seems like it's the cross-platform arena that gets the short-shrift when it comes to Apple's priority list. This is understandable, I suppose. We're the minority. Still, when basic things go wrong, Apple should hear about it, no matter how small the affected group may be. And they should fix it. Promptly.

So, if anyone out there in Mac-Land is having a problem with this, I say go vent. If you want to do it here in the comments section, by all means, be my guest. But you may want to take a moment and let Apple know about your problem as well (and trust me, they don't read this blog). There are a few good places to do this:

  1. Apple's Mac OS X Feedback Page
    Here is the best, and most appropriate, place to lodge a complaint or report a bug. Keep it business-like and polite, but specific and thorough. Vent someplace else.
  2. Apple's Discussion Forums
    Here you might actually find some help for your problem (though I think this particular problem is up to the Apple Developers at this point). Help notwithstanding, enough complaints on the discussion forum are likely to escalate the problem in the queue of stuff-needs-doin'. So it's worth a shot.
  3. Call AppleCare (1-800-275-2273)
    Again, a good place to get heard. It might be a pain. You might get put on hold. A lot. But if you've got the time -- and, of course, your machine is still under warranty -- this could escalate your cause up through the ranks of the Apple engineers. As always, be polite but firm. You may need to explain your problem to -- and humor -- a number of lower-tier help-desk types before you get someone knowlegeable. But it might pay off. Don't do this, though, unless you've got some time and patience. Trust me.
  4. Finally, spread the word
    That's what I'm trying to do with this post. Start writing to forums and blogs of some prominence. MacOSXHints is a good one to post on. There's also The Unofficial Apple Weblog. Hell, there are a ton, and they're all more prominent than this one. Start hitting them up. No one likes bad press. Least of all Apple. The more noise people make, the more likely they'll be heard.

So, anyway, hopefully this does somthing to draw a tiny bit of attention to an easily-overlooked, but non-trivial problem. I'm pushing for you kids. If anyone hears of a solution, please post it here in the comments. I'll do the same.

Now go start pestering.

MacOSX 10.4.3: What's Fixed and What's Not

I have to say that, so far, Mac OS X v.10.4.3 is a worthwhile upgrade. Actually, it's solved a good many of my main complaints thus far, and that ain't nothin'. Unfortunately, not everything is fixed. But what else is new? Here, now, for your perusal, is a list of problems, both fixed and unfixed. And mind you, this list is culled from my own personal set of experiences and complaints.

The Unfixed:
• The ever annoying Safari window bug is still present. Why won't this thing die? Why aren't more people bitching about it? This is major for me. I swore if they didn't fix it in 10.4.3, I'd switch for good, and damn it, that's what I'm going to do. Sure, Firefox is slower to launch, slower to load, and not quite as pretty. But you know what? It works. I don't care if it doesn't pass the Acid test. (Does anyone give two shits about this?) It works. On every page I go to. And that's what counts.
• The Finder's Inspector window, which is supposed to dynamically update its contents as you select files, still incorrectly shows the ownership when switching files, i.e. select a file owned by systemsboy, then select one owned by root. Everything in the Inspector updates, except the owner of the file. It still says the owner is systemsboy. To see the correct ownership, you have to close the Inspector and reopen it, which kind of defeats the purpose of the Inspector in the first place.
• I've recently been having a problem whereby switching between local and networked users with Fast User Switching sets my display profile to "Generic RGB Profile." Entering the Display Preferences pane and setting it back to my calibrated profile does not fix the problem. The only fix when this happens is to log everyone out, and log back in. This remains broken in 10.4.3.
• There is a bug whereby binding to a Panther Server is delayed for about 3-5 minutes after a reboot of a Tiger client. This means that, if you're a network user who's just rebooted a Tiger client, you'll get the shaking login window (as though you've typed the wrong password) until the Tiger Client can connect (bind) to the Panther Server. This persists in Tiger 10.4.3.

The Fixed:
• At last, some good news. One of my least favorite bugs has been corrected. Mail now properly handles tabs. You remember tabs, don't you? Those fixed-width whitespaces handy for table creation? Well, in previous version of Mail, they turned into plain old spaces (four, to be precise) right after you typed them. Now they behave properly. Yay!
• And good news on the Spotlight front: Spotlight responsiveness really is significantly improved in the new version. Typing is no longer delayed in the middle of pounding out a term. You can type as slowly of as fast as you want. It never gets stuck. Nice!
• Another Spotlight boon, for me anyway: Spotlight will now search the contents of files ending with the ".sh" suffix. As my primary use for Spotlight is to search my myriad shell scripts, almost all of which end in ".sh," this actually makes Spotlight potentially useful to me for the first time. Cool! (Yes, I have decided to end each of these bullets with an exclamation of some sort.)
-Update: Never mind. This is actually not true. There do appear to be some new options here, one called "Source Code" as a file type that I don't remember from before, but Spotlight does not appear to search its contents, only its name.
-Update 2: Scratch that last part as well. It's exactly the same in 10.4.2. Nothing's changed here.
• Another annoying bug that seems to have been cleared up (though I've thought this before and been wrong, so I won't swear by it) is that annoying scroll-wheel jump. You know the one I'm talking about: you're in your browser of choice, and you begin scrolling the page, ever-so-slowly, and WHAM! Next thing you know, you're at the bottom. Yeah, that seems to be fixed. Groovy!
• Finally, and this has huge implications for me and the lab where I work: Moving files across NFS filesystems (sub-volumes) which reside within another volume no longer writes zero byte files and (blessedly) works properly. This means we can go back to mounting user home accounts the proper way, with a simple automount call and a mount map. (For more details on this, you can read the full posts.) Sweet!

On a side note, as happy as this last bit makes me, I must say, I can't help being a bit cynical about it: We've had this problem for months now. I've spent months on it. And only in the last week or so have I been able, with much hard work and late nights, to come up with a workaround, and an elegant one at that (if I do say so myself). And now it's obsolete. As I wrote to a friend:

Mac OS X 10.4.3 fixes the FCP problems moving files across filesystems. Holy shit. One week after I come up with a workaround they release a fix. Is all of life this big a waste of time, or just systems work?

His response:

Just systems work...

I figured as much.

Oh well. I guess all my loginhook knowledge can be now put to good use elsewhere. Still, this seems like one of those things that might've been best fixed by just sitting on my ass and doing nothing but waiting (which I'm not very good at, apparently). And that leaves me feeling like I just wasted a whole bunch of time on this one.

Boo!

Anyway, this latest update doesn't fix everything, but it fixes a lot of important stuff on my list, and it doesn't seem to break anything so far (though TextEdit's "lists" feature did just start acting funny, and I don't know why). Was it worth the unendurably long wait? Well, I don't know about that. I sure wish the NFS problem had been fixed months ago, for obvious reasons. But I'd go so far as to say that, at least so far, as far as my experience goes, this update is a keeper, and, on first impressions at least, I highly recommend it.

Happy Halloween, kids. Now go forth and update. (Go on. I dare you.)

Happy Frickin' Halloween: Mac OS X 10.4.3 Arrives

Oh. My. God.

It's not on VersionTracker or MacUpdate. It's not on Apple's downloads site. There's no combo update as yet. But my Software Update just found Mac OS X 10.4.3. Finally.

I feel dizzy.

Okay, kids. Gotta run. Remember, you heard it here first.

UPDATE 1:
The combo update is out.

UPDATE 2:
Oh, Hallelujah! Just read this:
Pressing Tab while composing a Mail message now inserts a tab character, instead of four spaces.

UPDATE 3:
The 10.4.3 Server Update is also out now. Sweet!

UPDATE 4:
I've installed the update without incident. I will post a new entry soon about my observations on this release.

--------------------------------------------------------------------

From Software Update:

The 10.4.3 Update delivers overall improved reliability and compatibility for Mac OS X v10.4 and is recommended for all users.

It includes fixes for:

- AFP, SMB/CIFS, NFS and FTP network file services
- AirPort and Bluetooth wireless access
- Core Graphics, Core Audio, Core Image, and RAW camera support
- disc recording when creating and burning media
- .Mac sync services
- Spotlight indexing and searching
- Dashboard widgets: Dictionary, Flight Tracker, Stickies, and Unit Converter
- Address Book, AppleScript, Automator, Dictionary, Font Book, iCal, iSync, Mail, and Safari applications
- Disk Utility, Keychain Access, Migration Assistant, and Software Update
- compatibility with third party applications and devices
- previous standalone security updates

For detailed information on this Update, please visit this website: http://www.info.apple.com/kbnum/n301984

For detailed information on Security Updates, please visit this website: http://www.info.apple.com/kbnum/n61798