Tiger Hates Leopard

So I'm sitting in my office, waiting for the Mac OS X 10.5.2 Leopard update to show up in my Software Update, and it just isn't happening. And at some point I realize, "Hey, maybe it just hasn't been downloaded to our Software Update Server yet." Yes, we run our very own Software Update Server under Mac OS X Server 10.4. It's super cool. It downloads all Apple updates to itself, and then any Mac in our lab can get all the Software Updates from our own internal server, rather than Apple's, which just saves gobs of time and bandwidth. Oh, we love it. But there appears to be a catch: Tiger Server will not download Leopard updates.

So I'm sitting there waiting, like, forever. And the 10.5.2 update never shows. Nor does the Leopard Graphics Update, or the HP Printer Drivers Update, and I'm all like, "Dude, what the fuck?" When all of a sudden the iLife Support Update does show up.

And that's when it hits me: Tiger totally hates Leopard.

But that's okay, 'cause Tiger's a total bitch.

In any case, I'm not sure if the converse is true — if Leopard Server will fail to download and serve Tiger updates — but if it is, good luck running a Software Update Server in a mixed Tiger/Leopard environment. Geez! You'd think Apple's software would be more compatible with, um... itself!

Highly annoying.

Oh, and by the way, after removing my Leopard box from the SUServer client list, these commands got me back in business without a restart:

sudo killall DirectoryService

sudo dscacheutil -flushcache

Just so you know.

UPDATE:

It would appear that Leopard's Software Update Server is a bit less finicky when it comes to older updates, for previous versions of the OS. Hmmm... Do you get the feeling Apple's trying to tell us something? (You know, like, "Upgrade your server." Or something.)

Leopard Software Update Server: Backwards Compatible

(click image for larger view)

NetBoot Part 1

My big, fat, self-assigned new project — or, as I like to call it, the bug up my butt — is system roll-outs. That is, I realized at some point two things. One, I am managing way more computers than ever before, and way more than I realized; and two, the scope and variety of these various systems has become increasingly wide. These realizations inevitably brought me crashing, headlong (yes, headlong) into a third and final revelation: I need to come up with a better system for managing machine builds and systems roll-outs.

Enter: NetBoot.

Actually, let me first explain how we've handled this in the past. When I began this job we had maybe 15-20 Macs running OS 9 briefly, and then OS X since about its inception. Mac OS 9 was notoriously easy to build. Just copy that shit and be done with it. But Mac OS X was a different beast entirely. Mac OS X was complicated. Moody. A tougher nut to crack. Mac OS X required me to delve into the dark arts of system cloning.

Cue thunder clap, scary music.

The process that eventually evolved was cloning over firewire. We'd build our master machine — a basic Mac OS install with all the latest updates and our requisite software — and then clone that to the other machines over firewire, with clients booted into our Master via firewire target disk mode. This was a quick and dirty way to build a bunch of systems. Once the group was built we could customize machines or groups of machines as we saw fit. For a lab of 15-20 Macs this has worked swimmingly. But our lab has grown slowly and steadily, and completely without my realizing it.

My latest count shows our lab at more like 50 Macs now. We've added a bunch of stuff: laptops, servers, more A/Vmachines of various configurations, a render farm, and of course more workstations. Using our old system of firewire cloning is becoming increasingly clumsy, slow and error-prone. We need a better way of doing things.

Okay, now. Enter: NetBoot!

Leopard's NetBoot Interface<
(click image for larger view)

NetBoot is Mac OS X Server technology that allows for centralized storage of and access to system images for installation over the network. The way it works is this: You build a system, trick it out, make it perfect — this is your Master System. Put it on a firewire drive or something transportable, because you can't be booted from your Master System for the next series of steps — imaging. Run the application called System Image Utility, that comes with Mac OS X Server's server tools, and create a NetInstall image from your Master System. Load that image onto your server and enable it in the NetBoot settings in Server Admin. And what happens next is something akin to magic (unless you use Linux, and then it's pretty par for the course, I guess.)

The Network Boot Volume: So That's What It's For!>
(click image for larger view)

With your NetInstall image enabled on your server, go to any client system and open the Startup Disk Preferences. Where you'd normally see the "Network Startup" icon (I'll bet you always wondered what that was for), you'll now get something a bit more descriptive. You should now be given the opportunity to boot from your master image. Choosing to do so will incur further magical results. Your system will now boot... Over the network! What's even cooler is that you'll be booted into the same basic installer environment you'd see if you were booted off a Mac OS X install disc, and you'll be walked through the steps required to install your Master System onto your computer. Um... OVER THE NETWORK!

There are some immediate advantages to a system such as this. First off, you can have a bunch of different Master System images for various build configurations in your lab. For instance, we can have a separate build for laptops and desktop machines. Cool! Also, this can all be automated (thanks to Automator integration) to make the process run almost entirely unattended. Sweet! And since the whole thing sits on the server and is available at all times, if a machines needs a rebuild, you just set it and forget it. Awesome!

But NetBoot has its drawbacks as well. Images — particularly large images — take for-freaking-ever to build. To give you an idea of how long, my near-36GB boot drive took an hour or so to clone to firewire, then 3-4 hours to be imaged by System Image Utility. So each build will take several hours to create. And God help you if you make a mistake on one of your images: the NetInstall images are read-only and can't (to my knowledge) be modified once they're built. NetInstall technology can't be used for non-package installers or updates either (again, to my knowledge), so you'll have to run all your Adobe and Microsoft updates by hand as usual. Also, one minor caveat that threw me at the outset: Mac OS X 10.5 server can only serve Mac OS X 10.5 builds. So you'll either be needing to update everything to Leopard or wait until your server and client OSes match.

I think I'm right on the crux of really needing this. I could probably live without it. Keep doing what I'm doing. But I do think I'm at a point where the benefits of using NetBoot outweigh its limitations. And I have enough resources now (like the required massive drive space and robust stable servers) that it's not impractical on the physical level. So, this summer I plan to use NetBoot to build my lab.

I mean, imagine this: you get your new systems over the summer, you unbox them, plug them into your network, set them to NetBoot (everybody say "command-n"! Good!) and go home for the night. You come in the next day and everything's basically done. You've just built your lab. Overnight. In your sleep. I don't know. Sounds pretty cool to me.

I'll let you know how it goes.

Oh, and by the way, in case you can't tell, I'm just now learning all this. It's still very new to me and there is a lot about it I don't know. So if anyone has any experiences or insights into using NetBoot (or if I get any of the facts wrong), I would absolutely love to hear about it. Please feel free to share your experiences in the comments.

Duplicate Computer Names and IPs

So there's an incredibly annoying and puzzling behavior in Mac OS X with regards to duplicate computer names on the LAN. Most Mac Admins probably know what I'm talking about. Here's the deal: Let's say you have a Mac Pro on your home network named Spanky and it has an IP address of 192.168.1.25. Let's also say, for the purposes of argument, that your best friend — who likes to emulate you in every way — has a Macbook named Spanky with an IP of 192.168.1.25 on his home network. (Hey, it could happen.) Now let's say your pal decides to come over, and he decides, "Hey, I think I'll bring my Macbook over so we can swap some illegally obtained music and pornography." He gets to your house to find you happily surfing the 'net. He whips out his Spanky and plugs it into your network, fires that puppy up, and Bam! All of a sudden your Mac Pro locks up. You can no longer surf. And you get an error message that looks a little something like this:

Duplicate Computer Name Alert: Why Me?
(click image for larger view)

You go to your Sharing Preferences (or your Terminal, or what-have-you) and, sure enough, your computer has been renamed. Renamed! WTF! Why is your computer, on your network, suddenly called "Spanky-2"? Because that's how Mac OS X handles duplicate computer names on the same network. It renames the existing computer to existing computer-2. Not the intruder. Not the new kid on the block. Your computer is now the computer formerly known as your computer. Why, it's pure genius, I tells ya! Brilliant!

Seriously, what in God's name were they thinking? Because, the way I see it, this goes beyond annoying into the realm of the dangerous.

Case in point: Let's say your Mac Pro Spanky is actually a server that provides services — authentication, Kerberos, LDAP, file sharing, the works — for a network full of computers. And let's say your friend is actually a guest on that network. When that guest plugs his computer into your network, and it just happens to be named the same thing as your server, God help you. You just lost — well, I'm not sure how much, but — a significant portion of your services. And all it takes is a computer name? I can rename any Mac on any network from any other Mac on that network by just changing my Mac's name? What's more, if you can get the IP of that server, you can bring it down entirely. That is total shit.

Strangely, it's been this way for at least three iterations of the OS now — since 10.3 — and it's still this way in 10.5. I am appalled. Can someone please explain the rationale for this to me? Please? 'Cause from where I sit, this is a major security flaw. To my mind it makes way more sense to have the newly installed machine make changes to it's configuration than to essentially be able to force changes on another machine. It's just backwards. And dangerous. And it desperately needs to be fixed.

UPDATE 1:
Mat X points out two very important facts: 1) the name change in this instance should only affect the Bonjour (.local) name of the duplicate machine, not it's actual name (the name it calls itself) or it's FQDN (the name as resolved by a DNS server), and 2) a client name change on the LAN should not be able to bring down a server with the same name because of the previous fact.

Dude, you're totally right, though I have seen, way back in the old days (10.2 maybe?) a duplicate name kill a server. So it did used to be possible. Nowadays though, Mac OS X Server has better, smarter naming conventions that prevent such things. I will say, though, that what prompted this was that I booted a clone of my own machine up on the LAN (same name, same IP) and it killed my computer, internet-wise, probably more because of the duplicate IP address. It's possible that all would have righted itself over time if I'd waited to see. I was just so annoyed by the behavior that I went on a bit of a rant. I still think it would be better behavior to leave the existing LAN client alone and make changes only to the new Mac on the LAN. But still, I got a little carried away.

Thanks for keeping me on my extremely bitchy and unscientific toes!

UPDATE 2:
So I've been doing some more testing on this issue. And while duplicate names on the network will not bring down your Mac OS X Server, a duplicate IP address will. Here's what I did:

1. I changed my client's IP address to match that of the Mac OS X Server (both Leopard 10.5.1). My machine got kicked off the internet and I got this warning:
   
2. I rebooted my client machine. Once the client had rebooted, the server got the above alert.
3. At which point the server could not ping out to the internet. Bad Mac OS X! Bad!

So, while my rant is partially in error, there does seem to be a bit of a flaw in the way Mac OS X handles new duplicate clients — particularly duplicate IP addresses. And I maintain that a better way would be to only modify the behavior on the most recent addition to the LAN.

Leopard Beefs

So I really like Leopard. It's nifty, but not without it's problems. I thought I'd take a minute to post some of the issues I've had with the new OS.

So far I've only had a few complaints. Actually, my initial experiences were quite positive. I did an Archive and Install, preserving user and network settings, on my work machine, a Quad-Core Intel Xeon tower. I left all my old preferences in place and used my old home account data. All this went off almost completely without a hitch. The only snag, oddly, was my Final Cut Pro Studio suite, which needed to be re-serialized. Beyond that, smooth sailing.

Upgrade Woes
That upgrade went so well I decided to do the same to my aging PowerMac 2.7 GHz G5, and for the first time since I bought it that machine felt slow. Dog slow. Problematically slow. This was cause for concern. But what finally convinced me that there were major problems was what Leopard did to my iPhone. Oh, the horror!

Addresses
On the first iPhone sync with Leopard, all my addresses got completely borked. Each address was there, yet the name field was blank and there was a lot of information that didn't make the transfer. Fortunately I had a backup, and I was able to get back up and running. Oddly, the second sync worked perfectly, and only the latest contact info on my phone was lost.

Photos
Importing photos from the iPhone to iPhoto was also problematic on the upgraded Mac. The import took an extremely long time, and the imported photos suffered from strange color shifts and banding. They were unusable.

At this point I decided to do a fresh install. This worked much better, and my computer has been running much faster and handling iPhone syncs much more reliably. As part of the fresh install process, I also started with a fresh home account, which may have helped as well. But this was not the only source of problems, as I verified their existence on a clean account as well. Still, better safe than sorry, I figure.

In any case, for whatever reason, older hardware seems to like a fresh install. Things have been working reliably now for a few weeks. I'm happy.

AFP and MPD
My other beef with Leopard has to do with AFP mounts in the Finder. The way the Mac OS has always dealt with such mounts in the past is that when you mount a shared drive, you authenticate as a user with access to the resource and the drive mounts as though the authenticated user had mounted it, even if that user is different that the one who's logged in. So, for example, I could be logged in as SystemsBoy, but authenticate to a server as JimminyCricket. The server will show me the available shares for JimminyCricket, not SystemsBoy, even if they have different access privileges to that server. If I unmount the share, I can then reconnect to the server, this time authenticating as SystemsBoy, and then I see the shares available to SystemsBoy. This is actually more useful than it sounds. I have numerous identities that I use on my network. This allows me, for instance, to be logged into a machine as a non-admin user, but still connect to a server as an admin user and have admin access to those resources.

In my initial experiences with Leopard, however, the behavior had changed in a way that was potentially sucky. In Leopard, when I'd connect to a share I'd authenticate as a user, just as in the past. And the share treated me like the authenticated user, just as in the past. But once authentication had taken place, Leopard remembered the user credentials even after ejecting the mount. At this point, when attempting to re-log in as a different user, the Finder would refuse to forget the previous login identity and authentication would be bypassed. The server would simply remount as the originally logged in user.

This behavior could be a convenience for users who only use a single identity on client and server. But for those of us with MPD (that's Multiple Personality Disorder, smart guy) it's a real problem. We'd now find ourselves unable change our login identity on a server without a reboot, or until some period of time has expired. Crap!

Now, I have to say, this was happening to me without fail a few weeks ago, and it was really annoying. But now, for some odd reason, the behavior seems to have reverted back to it's old self, asking for authentication each time I request a server. It's really quite strange. I know this was a problem, because I noted it in my log of Leopard issues, but I'll be damned if I can reproduce it today. I suppose it's something to watch out for, but I'm glad to see that it seems to have cured itself. In any case, clearly there are bugs in Leopard, however inconsistent and occasional they might be.

~/Library
Leopard is a bit of a hard ass when it comes to your Library folder. The Library folder is where all the user's settings are stored, and Leopard takes steps to insure that it's always there. In fact, it disallows you from renaming or moving the folder. Try to rename it and it simply won't highlight; try to move it and you can only copy. This is both a blessing and a curse, I suppose. I mean, from an admin standpoint, I never really have to worry again about users removing their ~/Library folders. But then again, I never really did anyway. Conversely, it requires a trip to the Terminal to move a user's Library folder and test any problems therein, which is a fairly common troubleshooting step. I guess we break even here, but I think I preferred the old way of doing things. It just made my life easier.

Uh... So far that's it. Compared to my Tiger Beefs from a couple years back, that's nothin'. Hell, compared to any new software release that's frickin' amazing.

I'll say it again: Leopard is a pretty damn sweet release.

UPDATE:
I continue to have AFP problems on my Intel machine. They're different from those detailed above, but no less annoying. Earlier, I attempted to connect to a server and the process hung. I had to force quit the Finder to get out of it. And just a few minutes ago I connected to a server and copied a group of files over to my local system using a "command-c" copy, only to be denied for permissions reasons. Dragging and dropping the same batch of folders worked properly. And now, after ejecting and reconnecting to the same share, both styles of copying the same, exact folder work just fine. So clearly there are some AFP bugs that need to be worked out in Leopard. Nothing life-threatening, but surely annoying.

Leopard Quota Alerts

Our home account server — the one all our network users' home accounts live on — mounts via NFS. I've already mentioned one of the improvements with regards to this behavior in Leopard, namely, autofs. But I just discovered another.

See, that NFS mount I'm always talking about, well, it's a Linux RAID — Fedora Core 6, I believe. And we've put quotas on every user's account to keep it from filling up. This has worked great, for the most part. The problem has always been how the Tiger Finder handled a user exceeding his quota. Basically, what would happen is that any Finder copy that exceeded a user's quota would fail, mid-copy, with an extremely vague and essentially — unless you happened to know you were looking for a quota problem — useless error message. Can't recall it exactly, but it's something along the lines of: "The Finder encountered an error and could not complete the request."

Ew.

The good news is that the Leopard Finder behaves exactly like we'd want and expect:

Leopard Quota Alert: That's More Like It!
(click image for larger view)

This alert actually came somewhat prematurely, as I hadn't actually exceeded my quota, and the alert seems to be triggered by hitting the soft quota rather than the hard. What's up with that?

Still, I have to say, I'm endlessly impressed with these little tiny details. They make all the difference in the world from an admin standpoint. I no longer have to worry about quota weirdness! It's great to see Apple do so much for admins with this release. I'm seriously pleased as punch.