More Leopard SysAdmin Goodness

The other day a staff member's firewire drive tanked. I could see it in Disk Utility, but I could not repair it, nor could I mount it. It was in a bad way.

After exhausting my standard barrage of tests and utilities, I remembered that Leopard seemed to have some rather advanced disk management capabilities not found in previous versions of the OS. So I figured I might as well try examining the bad drive on my Leopard install, if for no other reason than to see what happened.

What happened was awesome. Upon attaching the drive to my Leopard install I received this alert (note the drop-shadow, which is now included in Leopard window caps):

Leopard Disk Repair Alert

(click image for larger view)

From there the drive mounted read-only, and I was able to copy most of the data from the damaged drive. Initially I was copying from the Finder. The problem with this method is that, at least in this iteration (build 9A527), every time the Finder hits a corrupt piece of data, the copy stops — hangs, actually — and you have to force quit the Finder and pick up the copy where you left off. Painful if you have lots of broken bits. In the end I used the ditto command to copy over the data. The ditto command will continue copying even after hitting bad data, and it will report which data is bad as well. I sent the output of ditto to a text file and so was able to run the process unattended while still having a record of which files were broken. Pretty sweet! I feel smart!

If you're wondering, my command looked like this:

sudo ditto -V -rsrc /source/directory /target/directory 2> DittoOutput.txt

But the real news here is how cleverly and usefully Leopard seems to be dealing with disks. And how Leopard has some genuine new goodness for SysAdmins. That's pretty amazing. I'm getting excited.

I'm not sure this post violates my NDC. It probably does. But honestly, I've seen other sites posting much more revealing stuff, so I'm going to assume I can probably get away with this. If not, well, we'll know soon enough. But this is just too cool to keep secret.

Apple Remote Desktop Copy Problems

Prosaic title, I know. But it's true. Every now and then, copying to remote systems via Apple Remote Desktop fails inexplicably. Fortunately, the solution is a simple one: restart the ARD Agent, conveniently named ARDAgent. Said command will look something like this:

sudo killall -HUP ARDAgent

Ironically, you can also send the command to the offending system via ARD itself. Just be sure you remove the sudo and send it as root.

Restarting ARD Via ARD

(click image for larger view)

That's it! Just another helpful tip from your friendly neighborhood Systems Boy.

Please resume your normal activities.

I Was Right

Well, I can't say too much because I'm under non-disclosure and I'm not sure what's fair game to discuss, but the Leopard client I've been playing with has some geekably great features — stuff I've been waiting for Apple to make easy for a long time. One of those features is something I alluded to a long time ago, when Boot Camp first hit the scene. I'd said:

...I remain hopeful that this utility is indicative of possible future utilities that would allow for more flexible, less destructive partitioning.

While I can't really say anything definitive, I will say this: looks like I was right.

Neat-O!

"Disable Clear Text Passwords" Breaks Randomly

Typically, when you set up a Mac client to bind to a Mac server using Directory Access, there is one lone entry in the Security settings that is checked by default, and that is the "Disable clear text passwords" setting. This seems like a prudent default, and I always leave it checked. I assume that this means that passwords are then sent to the authentication server in some sort of hashed or encrypted form, and that both server and client are set up to negotiate this transaction properly out of the box. Indeed, most of the time this does not present any sort of problem whatsoever.

But for some reason, every now and then, completely randomly, Mac clients will suddenly and mysteriously be unable to authenticate to my Master Authentication Server. Seriously, nothing's changed. Just all of a sudden, Macs can't authenticate. The solution? Un-tick that "Disable clear text passwords" box under the LDAPv3 server configuration's Security tab. Next thing you know, everything's right as rain.


Directory Access: LDAPv3->Select Configuration->Edit
How in Hell Does this Break?

(click image for larger view)

Seriously, can anyone tell me what I'm doing wrong? 'Cause frankly, it's annoying.

External Network Unification Part 5: Almost There

It's been quite some time since I've been able to post anything of any substance. This has a lot to do with the fact that I've been super busy relocating our department and participating in the gut renovation of our lab. This has been an immensely stressful process, but in the end I find that I've learned so much from it, I simply can't complain. I'm coming out a far better SysAdmin than I was going in. And that's a remarkably valuable thing to both me and my employers.

But moving and planning the physical aspects of the new lab has only been a portion of what I've been working on. This renovation has been the perfect opportunity to rebuild our network infrastructure, and part of said rebuilding has resulted in the near completion of our authentication unification project. At this point we've gone from eight different authentication servers — that is, anytime we created a new user, we had to do so on eight different systems — all the way on down to two. Which means that now, anytime we create a new user, we do so on two machines.

Our goal is to get it down to one, hopefully before the Fall semester begins. Our mail server is proving to be the most difficult machine to get working with LDAP authentication, mainly because it authenticates mail users through the wonders of some weird combination of authd, Courier and PAM, and we've yet to crack the magical code that gets these all working in tandem via LDAP. Aside from Mail, though, everything is done. So I thought I'd take a bit of my hard-earned vacation and loosely describe to you how it's all working.

Before I start I'd like to just acknowledge all the help I've had from my fellow SysAdmins in the department. I had a huge amount of assistance on the *NIX server side of things, as well as with network infrastructure and even some last-minute PHP finagling without which this project would have taken significantly longer. In fact, all I really had to do was build the authentication servers and clearly articulate what I wanted. I'm extremely grateful to everyone who helped out.

The little bit of network infrastructure I mentioned is our DMZ. We now have a proper — and more importantly, properly secured — DMZ on which to place an authentication server. I won't go into too much detail here, but suffice to say, having a secure DMZ gives us all kinds of options for authentication between internal and external networks, and makes me feel a whole lot better about using Mac OS X Server as our authentication system for both networks.

Yes, we are using Mac OS X Server to authenticate our entire network. The reason is because Mac OS X Server is the most mature and usable implementation of LDAP for user authentication available on the market today. Is it perfect? No. Is it completely secure? Probably not. Is there anything that even comes remotely close to being able to handle the complexities of user management and database redundancy across platforms with such remarkable ease-of-use? Nope. Nothing. We tried building our own custom LDAP server, which would have been excruciating, and would have taken forever. We tried Red Hat's Directory Server, which looks like it will eventually turn into something to match Mac OS X Server, but which just wasn't yet up to snuff. Nothing matched Mac OS X Server, which did everything we wanted it to, right out of the box and with a minimum of fuss. In fact, once the user database is built, building a Mac OS X master or replica authentication server is a complete and total breeze. At the time of our building and testing it was really the only practical option.

So, here in a nutshell, is what we have:

Internal Network
All authentication originates from the internal network. Passwords can only be changed from the internal network at this time, which is by design. Systems on the internal network include:

  • Master Authentication Server
    Hosts authentication for... Well... Everything, really. This is essentially the same server we used all last year for all our internal authentication needs for Mac, Linux and Windows workstations. It's now being used to push authentication to the external network as well.
  • Internal Replica Authentication Server
    This provides replication of the Master. Should the master fail, the Replica is intended to pick up services (though this doesn't always work perfectly).
  • File Servers
    We have two file servers on the internal network — a Mac and a Linux box — both of which authenticate directly against the Master.
  • Workstations
    We have about 30 Mac, Windows and Linux machines all authenticating to the Master.

DMZ
The DMZ sits between the Big Bad Internet (BBI) and the internal network. It has its own firewall that is fairly strict about what can get in from the BBI. All DMZ authentication originates from the internal network, but is provided by a single server which sits on the DMZ. Systems on the DMZ include:

  • External Authentication Server
    This server is also a replica of our Master, but it's not intended as a failsafe. Rather, it provides authentication services to the entire DMZ. It gets its user database, of course, from the Master. But for other systems to bind to an LDAP server, its role must either be "Master" or "Replica." Setting the role to "Connected to a Directory Server" won't work. In addition to sitting on our DMZ, which is properly firewalled against the harsh realities of the Big Bad Internet (BBI), this system also makes use of its own strict local firewall for an extra added layer of security. Also, all replication communication between Replica (DMZ) and Master (Internal) is encrypted.
  • Data Server
    In addition to unifying authentication, we've also consolidated data storage and access wherever possible. In the past, for instance, movies streamed from the Quicktime Server were stored on that machine's local drive. Web sites were stored on our web server. So, building a web site that used Quicktime Streaming required users to log into two separate machines — the Web Server and the Quicktime Streaming Server. Now we're storing all user-generated content on a separate, dedicated machine — our Data Server — and sharing that machine out to the various servers via NFS. Centralizing this data store means users have only to log on to one server for anything they ever want to do. And also that only that server needs to authenticate users. And yes, that server authenticates them via LDAP on our External Authentication Server. All neat and tidy. Internal and external home account data is still segregated, however — users still have separate internal and external data storage. Though, if we could figure out how to do it securely, this could change.
  • Quicktime Streaming Server
    This machine also uses its own local firewall. It gets its user database from our External Authentication Server over secure channels using the "Connected to a Directory System" as its role currently. Ultimately, however, because of the Data Server, this machine will not need to authenticate users. We are leaving the ability open temporarily to accommodate legacy users.
  • Drupal CMS
    Our new Community site is built on the Drupal engine. We're using the LDAP module to authenticate to the External Authentication Server. Drupal's LDAP module is simple and easy to set up, as is the Drupal system as a whole. So far we're very happy with it.
  • Computer Reservations System
    This is a custom web app built long ago by a former student. We've (and by "we" I mean my colleague) basically hacked the PHP code to authenticatevia LDAP rather than MySQL.
  • Mail Server
    Currently not authenticating to the External Authentication Server. We're working on this and hope to have it working by the beginning of the school year.

The Future
Yes, there's more we want to do. It's always amazing how, once you've completed something, you immediately start seeing ways to make it better.

  • More Redundancy
    Ultimately, in addition to the Replica, I would also like to automate a clone of the Master's boot drive to an external firewire drive as sort of an ultimate safety. Should anything ever go wrong with the Master, I simply plug the firewire clone into virtually any Mac system on the internal network and I'm back on my feet. It might also be wise to have some sort of failsafe for external authentication as well.
  • More Security
    while our setup is fairly secure right now, there are a few areas I'd like to beef up even more when I get a chance. In particular, our CMS connection is not as secure as I'd like it to be. And ultimately I'd like to harden every machine on the DMZ to the best of my ability.
  • More Unification
    Anything else we can unify — and at this point that's mostly internal and external data — I'm open to considering. It's going to be really interesting for me to look critically at what we've done so far and find the flaws and refine the system. But I'll constantly be looking at ways to simplify our current setup even further without compromising security. The easier our network is to use, the more useful it becomes. We've come a long way, but I'm sure we can find even better ways to do things.
  • More Services
    Now that we have an infrastructure in place for user creation, we can add services freely to our network without the worry of creating users for said services. New services need only the ability to authenticate via LDAP. We're already planning an equipment checkout system, and possibly some calendaring systems.

So, I've just finalized the master authentication server. It's done. Built. Finished. Kaput. The rest of our servers are still in various states of finality, and we have until September to lock them down. But right now, unified authentication is, for all intents and purposes (and with the exception of mail), working. And we couldn't be happier. The ultimate test will be, of course, letting users loose on this new infrastructure. I'm betting they'll like it almost as much as we do. At least the ones who know the old system. New users will be none the wiser. Ain't that always the way?

*Sigh*