"Go Away..." Redux: Simplifying the Complex

Well, it's official. The "Go Away or I Shall Replace You with a Very Small Shell Script" post is the runaway hit of the season. It's the most popular — or at least the most commented on — post on this site since "Getting Back to (Search) Basics," which is odd considering how different the two articles are: One is a simple, utilitarian shell script; one is an opinionated rant. Strange what strikes a nerve. But that's not really what this post is about.

In the "Go Away..." post I postulated, in essence, that perhaps this latest generation of users seems to be less tech-savvy than mine because they've had technology handed to them on a silver platter. I was pissed off and in my annual funk, or bitch-mode, or whatever the Hell it is that makes me so irritable 'round this time of year, and I needed to vent, and my logic skills were not at their finest. My brilliant theory may have been somewhat half-baked (although I still think it's kind of interesting). In the comments, people either agreed with me or didn't, and if you disagreed with me and I flamed you in any way, I want you to know that, though I'm not exactly apologizing for it, your comments are truly appreciated. I try to respond to every comment on the site, because responding forces me to rethink and restate my position on things, and that tends to get my brain a-workin' and that's always good (even when it keeps me writing 'til 4 AM). So on thinking about my theory in "Go Away..." and the the comments therein, some things occurred to me that I'd not considered in the original rant, and these lead down some interesting systems-philisophical roads.

The basic idea that I want to talk about here is simplicity. I realized that as time has gone on, maybe users are less tech savvy not because they're jaded and spoiled — no, maybe they haven't changed at all. Maybe it's because technology has grown so much more complex. I know that in our lab this is the case. When I first started this job, nearly six years ago, we were using Mac OS 9. Overall, as you probably know, OS 9 was a far less capable OS in terms of network capabilities, multi-user capabilities and reliability. But, man, was it easy to administer! Mac OS X is vastly more useful — has vastly more capabilities — but it's also infinitely more complex, and therefore more difficult to setup and maintain. Apple has done a great job of making OS X almost as easy to use as OS 9, from a user standpoint, but the simple addition of a multi-user environment, for instance, (not to mention the UNIX layer, networked home accounts, permissions and the like) has made it more difficult to use than previous Mac operating systems. And this additional complexity has spread throughout the entire lab, thus making the lab itself more difficult to understand and use. I once had a girlfriend who preferred McDonald's to Wendy's because the Wendy's menu offered too many options. In a complex world there is a certain appeal to simplicity.

Apple has done an absolutely amazing job in the realm of simplification. Mac OS X, as incredibly complex as it is, is still the easiest OS in the world to use. Mac hardware — even the pro hardware — is unbelievably easy to set up. And the unmitigated success of the iPod over literally all its competition — even cheaper, more feature-laden players — is proof positive that simple sells, and that Apple is the master of this philosophy. I first understood this when Safari was introduced a few short years ago. Here was a browser from which they'd stripped out numerous features, but which (after the release of the tab-enabled version) I found myself using as my primary browser. Though I did recently switch, for certain unavoidable professional reasons, I still prefer the Safari experience to that of any other browser. And part of the reason is because of its simplicity. Safari is a joy, in large part because it's simple and easy as Hell to use. You find this approach in every Apple product. It's a hallmark of the brand, and it's the reason Mac users are so loyal.

When I started this job I was a Macintosh SysAdmin. Mac only, in a mid-sized, very cross-platform networked lab with a separate SysAdmin for each platform. Over time we've restructured the staff, and in recent months responsibility for the whole lab has suddenly come under my purview. I now spend a lot of time thinking about how the lab is currently constructed and how I want it to be in the future. Right now, things work pretty well overall, and our network is quite powerful. There's a lot a user can do to leverage the power of this network if they have some understanding of how it works. The problem is, there's no way they can possibly understand how it works with their limited time and access. In "Go Away..." I said that, to a certain extent, people need to understand the tools of their trade, and I stand by this. But how well do they need to understand? How deep should their knowledge run? Should the car mechanic have a complete and thorough understanding of the way each and every individual car part is machined or manufactured? Should the car owner? Probably not. And I would argue that users of our lab should not need to have much understanding of the nuts and bolts of its construction in order to make use of the majority of its power. The functionality of our lab, like with an Apple product, should be almost self-explainatory. This is what I think about when I think about how I want our lab to be: How do I make the complex simple?

I look to Apple for ideas along these lines. How does Apple streamline and simplify the use of a product or a piece of software? There are a few ways. Firstly, they remove the unnecessary. If it's something people aren't using, they strip it out or they hide it. Apple is really good at deciding what the most important features of a product are and stripping out — or at least hiding to a certain degree — everything else. Everything you need to know is right there in front and working as 99% of people would expect, and more advanced functionality is hidden away in places where more advanced users know to look. So that's step one: remove the cruft. Take out everything we don't need and present a limited set of the most essential options.

Step two is what I guess I'd call sensibility: presenting things in a logical and intuitive manner. Once we've boiled functions down to the essentials, how do we present them to the user in a straightforward and easily understandable way? This step is hard because it requires a good understanding of users in general, and of our particular user base specifically. How do our users tend to use the computers in our lab? A great way to find this out is to ask them. I try, when I'm implementing a new feature — or updating an existing one — to ask a sample of users how they'd use the feature, or how they'd expect it to function. I also look to Apple again for inspiration. How does this feature (or a similar one) work in a Mac product? Why did they choose this method? What about it makes sense? If you can get a sense of how users are using your lab and the computers therein, you can then implement new features in a way that's easy for them to learn and remember.

Step three is consistency. Final Cut Pro is a great example of Apple's consistency. I've been using FCP since version 1. We're on version 5 now, and yet I've almost never had to relearn how to do anything in the program. With all the myriad new functions added in the last seven or eight years, almost every command functions exactly the same way it did in version 1. The interface layout is almost the same in every way. Apple may add features, but they're insanely consistent when it comes to maintaining usability from version to version of FCP. It's this kind of consistency that makes continued use of Apple products a very organic and easy affair. Similarly, in the lab, o nce we implement a new feature in a sensible way, maintaining the way that feature works from semester to semester should be as consistent as possible. This is not always easy, and sometimes even Apple themselves are to blame for broken consistency. God knows, differences even between Panther and Tiger have complicated the consistency problem for me and my users, and there's not a lot I can do about this sort of thing. But as long as general consistency is considered and maintained whenever possible, and combined with sensible implementations, it goes a long way to simplifying use and creating an environment that "Just Works." An example of where sensibility and consistency have broken down in our lab is the numerous password servers we employ. We have seven or eight (I can't even remember anymore) different password servers for the various platforms and web application on out intranet. This means that passwords for Macs are stored on a separate server than those for Windows or Linux or any of the other authenticated systems in our lab. This is neither sensible nor consistent. Users' passwords match at the outset of their enrollment here, but a change on any one platform will render the set of passwords inconsistent between platforms, and this is extremely confusing to users. I've spent untold hours working on (and writing about) this problem and plan to implement at least part of my solution this summer. Suffice to say, inconsistency in user data and workflow does little to simplify lab operations, and does a great deal to complicate them. And this makes the lab much harder to use than it should be.

The final step in simplifying the lab is education. Once intuitive, sensible, consistent features are implemented, we need to tell people how they work. Fortunately, if your implementations are truly intuitive, sensible and consistent, this shouldn't be that difficult, and should be something you don't have to do over and over again. Instructions on how to use the lab should be simple, clear and to the point. They should be illustrated whenever possible. And they should be documented online somewhere that is intuitively, sensibly and consistently accessible. Relearning the network should be as easy as opening and skimming a web page.

What we're really talking about here is the user experience. I think, personally, my job as a SysAdmin and Lab Administrator, at its heart, is really all about creating a user experience. Right now, in many ways, the user experience in this lab is more akin to Windows than it is to Mac OS X. There are too many steps to do simple tasks, things don't work as you'd expect, and there's a lot of confusion about how things work. Hopefully, it's getting better. Unfortunately, there is probably no way for me to make every change I want to make in one fell swoop. Changes to our existing infrastructure need to made, to some extent, with legacy users in mind. Plus there just isn't time to plan and implement everything at once, nor would it necessarily be wise to do so. So we move in baby steps. Occasionally leaps and bounds. But always piecemeal. In the end we hope to have a vastly simplified and, as such, vastly improved user experience that allows our users to focus less on how our lab is set up behind the scenes and more on how they can use it to do their work. That is what I see as "my job."

Thanks again to readers and commentors on this site. You've given me a lot to think about and I appreciate it.

UPDATE:
Another blogger, software developer Daniel Jalkut, posts some similar (and similarly long-winded) thoughts on simplification from a software development perspective on his Red Sweater Blog.

UPDATE 2:
Yet more. Seems to be a hot topic these days. (Via Daring Fireball.)

Apple Software RAID Tests or: What to Do if You Don't Have an Intel Mac

Yes, I got very excited about what I saw in Boot Camp, though for maybe different reasons than most, and now I'm over it. I don't have an Intel Mac, so I couldn't really do much with the Boot Camp Assistant, but I did get turned on to the latest software RAID capabilities included with the Mac OS. Specifically, two items I was previously unaware of: concatenated RAID and the new resizeVolume verb included with the latest version of the diskutil command. Since I promised I'd report about anything I found, and since I did take the time to perform a battery of tests on all things Apple Software RAID, I thought I'd share what I found, though there's not much here that's very exciting, or perhaps even very enlightening. Still, what follows is a fairly thorough explanation of the various RAID options available in Mac OSX. For PPC, that is.

First of all, software RAIDs cannot be partitioned, which I actually didn't realize, though it does make sense.

Attempting to Partition a Software RAID: No Go

(click for larger view)

The two most common and useful types of RAID are Striped and Mirrored. A RAID stripe combines two drives for increased storage and performance. A mirror combines two drives for redundancy. Here are some details on creating RAID stripes and mirrors on Mac OS X.

Striped RAID

  • Increase in capacity
  • Increase in performance
  • No redundancy
  • Works best with disks of equal size/type
  • Can be used with disks of different sizes
    • Consequences unknown
  • Deletes disk contents on creation

Mirrored RAID

  • Half capacity of combined disks
  • No performance increase
  • Redundancy for disk failures
  • Works best with disks of equal size/type
  • Can be used with disks of different sizes
    • Total size/redundancy is equal to the smaller of the two members
    • Other consequences unknown
  • Deletes disk contents on creation

Most of this I already knew, though I did discover that the RAID mirror can also be set to automatically rebuild in the event that one of the disks fails. Unfortunately, there is no notification for disk failure. So you can automatically rebuild the set, but you have to manually check its status. This is obviously putting the cart before the horse. Kind of stupid. But whatever.

RAID Mirror AutoRebuild: I Didn't Know it was Broken

(click for larger view)

What I did not already know about, however, was something called concatenated RAID. Turns out concatenated RAID (or "JBOD" for Just a Bunch Of Disks) is the oldest form of RAID, and has generally fallen out of fashion. No one uses concatenated RAID anymore. With the current size of hard drives, there's little reason to.

Basically, concatenated RAID does only one thing. It combines a set of drives for increased capacity. There is no performance increase. Concatenated RAID can be used with combinations of drives of any size and can be added to dynamically. If you add a disk to a concatenated RAID, the added disk is erased but the contents of the RAID remain untouched. There is also no redundancy with concatenated RAID, so if you lose one drive in the set, you lose all the data on the RAID. Here's how concatenated RAID works in Apple's Disk Utility.

Creating a Concatenated RAID Set: Data Will be Deleted

(click for larger view)

Concatenated RAID

  • Increase in capacity
  • No increase in performance or redundancy
  • Works with any set of disks
  • WILL delete disk contents on creation
  • Unmounting one member of the set will cause the RAID to fail completely; RAID must be mounted/unmounted all at once
    • Rebooting fixes the broken RAID
  • Adding to the set deletes data on the new disk, but keeps data on the existing set
Updating a Concatenated RAID Set: RAID Data is Preserved

(click for larger view)

Resizing Partitions with diskutil resizeVolumes

Finally, in Mac OS X 10.4.6 there is a new option for diskutil, the command-line version of the Disk Utility application. This new verb is called resizeVolumes and is intended for doing just what it says: taking existing partitions and resizing them dynamically. Unfortunately, as indicated in the comments of this MacGeekery article, though the option is included in the PPC version of 10.4.6, it only works on GPT (GUID Partition Table) formatted partitions which are not normally created by PPC Macs. PPC Macs use the APM (Apple Partition Map) format for their partitions, so on my machine the command failed with the following errors:

Attempting to Resize an APM Volume on PPC: Fat Chance!

(click for larger view)

This appears to be more than just a partition format problem though. The new Disk Utility in 10.4.6 allows for creating partitions with the GUID scheme:

Choosing a Partition Scheme: No Help for the Intel-Challenged

(click for larger view)

But attempting to resize these with diskutil still fails thusly:

Attempting to Resize a GPT Volume on PPC: Nice Try!

(click for larger view)

So, it would appear that, although there are some intriguing new partitioning tidbits in Mac OS X 10.4.6, they are not to be enjoyed by the Intel Mac-less. Alas! Such is life on an educational salary. If I can convince someone to buy me an Intel Mac, I'll surely mess around with this stuff some more. Until then I guess we'll leave the real cutting edge stuff up to the early adopters.

Lucky bastards!

Should Labs be Dual-Boot?

With today's official arrival of Windows-on-Mac, the question has already been posed, both online and to me personally. All day long people have been emailing and stopping by my office to talk about this development. It's truly incredible. And, of course, one person even suggested, as I knew someone would, that we purchase all Macs for the lab and make everything dual-boot Mac OS X and Windows. This is not the first time something like this has been proposed. In fact recently students were clamoring for dual-boot Linux/Windows systems (though what they really wanted was more Windows and fewer Linux systems). This issue just keeps coming up, and at this point I'm starting to wonder if it's not a terrible idea, and how to best implement such a thing.

My arguments in the past were essentially threefold: 1) Users who are less technically inclined would be put off by dual-boot systems, and we're really trying to make things simpler around here, so complicating the boot process was maybe not the way to go; 2) Building two systems for every computer equals twice the work; 3) Maintenance and troubleshooting system problems becomes exponentially more difficult. Let's address these point by point.

Inexperienced User Issues
I worry that new or inexperienced users would find the dual-boot process intimidating. While I still think this holds true to some extent, I must admit the bootloader provided by Apple looks astoundingly easy. Downright appealing in fact. I would expect nothing less from Apple. I think this minimizes, but does not negate, the difficulty factor for new or less experienced users. But I'm also on a big push here in this lab to simplify the way things work in general (in the grand Apple tradition). Currently everything from logging in to file sharing is just too complicated. And needlessly so. Making all our systems dual-boot would only add to the confusion. So if we ever added this wrinkle, I'd want to vastly simplify what we currently have first.

Twice the OSes Equals Twice the Work
Any computer that is dual-boot must have both Windows and Mac operating systems installed on it. Each OS must also have a full suite of applications installed. Thus, building such a system would require twice as much work as its single-boot counterpart. If there were a way to build one dual-boot system and then clone it to our other computers it would greatly simplify things. But something tells me that it won't be that easy. I have a feeling the Mac will not be able to clone the Windows partition, making rollouts an absolute nightmare. If it turns out not to be so bad, it will do a lot for recommending the dual-boot paradigm. But we shall see.

Maintenance and Troubleshooting Issues
A dual-boot system is not only twice as much work to build, it's also twice as much work to maintain. Any update that needs to be applied, be it system or software, must be done from the appropriate OS. So if a system is booted in Windows, and I have Mac updates to perform, I have to physically walk over to that system and boot it into the Mac OS before I can perform the update. Any system or hardware problem the computer might have must also go through additional new layers of troubleshooting. And if the fix requires any sort of wiping of the hard drive, restoration of the affected system will be twice as difficult as it would on a single-boot machine. Not to mention the fact that, although this probably wouldn't be the case, it is possible that a dual-boot system, because of its complexity, would be less stable than a single-boot computer.

Pros and Cons
So I ask, should the lab be dual-boot? After thinking about the above points I would answer no. At least not our lab, and at least not yet. There are situations in which multi-tasking scenarios are desirable, but there are also situations in which dedicated systems are preferable. In our lab we have a pretty good balance of dedicated Windows and Mac systems. We have enough of each system so that every user gets enough time in his or her operating system of choice. Systems are continually in use, yet no one is fighting for resources. We don't need more Windows machines, and we don't need more Macs. So the only advantage I can see to a dual-boot lab is the convenience of choosing your OS from any computer. The potential drawbacks are many, however: greater confusion, system instability and significantly more work for our systems staff, to name a few.

I can see situations in which dual-boot machines would be a huge windfall. Specifically, if you have a limited computer budget, you now have the capability of getting twice the bang for your hardware buck with an Intel Mac. Every computer can essentially be two computers, as long as it's an Intel Mac. But our computer budget is not limited, and there may still be good arguments for using non-Mac hardware for a lot of the things we do here. So I don't see lab-wide dual-boot systems for our lab anytime in the near future.

But then again, you never know.

UPDATE:
Looks like I'm not alone. Other lab administrators chime in. Here's a choice quote from rambleon.org's Jay Young that sums up exactly how I feel about all this from a sysadmin point of view:

"It’s hard enough for any one of us to keep one Operating System maintained, let alone two. Especially when you have to restart the whole system to get to the other one."

Right on, brother!

Boot Camp and Partitioning: Cooler than You Think?

Today Apple released a beta software package called Boot Camp to facilitate running Windows XP on an Intel-based Mac. I'm practically speechless. Which doesn't happen too often.

Apple's Boot Camp Beta: Brilliant!
(click for larger view)

Now we all know a few weeks ago this nut was already cracked by some enterprising young hackers, so we all knew it was possible, and in fact probable that installing and booting Windows on Mac would become a fairly trivial and commonplace happening. There were even rumors that the next version of the Mac OS would provide "virtualization" software for running Windows in an emulated environment from within Mac OSX. But I have to admit I was quite taken aback by today's news. Official support from Apple for installing and running Windows on the Mac.

Brilliant!

Dual Booting
(click for larger view)

One of the best things about all this is the fact that Apple is providing Windows drivers for Mac-specific hardware. So you no longer need to search the internet to find them. But to me, coolest thing is the portion of the assistant that creates the partitions. Here, for the first time in Mac OS history (at least that I'm aware of), we see an Apple-branded utility that will partition your drive without erasing it. It's uber-smart of Apple to provide this functionality for people who want to test this out without the inconvenience of a full system reinstall. But it also has immensely cool implications if partitions are as important to you as they are to me, or if you happen to use an Xserve RAID. I'll explain.


Space Maker: The Caption Says it All
(click for larger view)

I use partitions on my systems to wall-off user data from system data. So I have one partition with all the system components and applications. Root. The boot drive. I try to keep this drive to as reasonable a size as I think I'll ever need, and then create a second partition with my user data: projects, mail, videos, music — everything else, essentially. This partitioning scheme works great, but it breaks down when someone like Apple decides to release a suite of applications that suddenly takes up eight times more space than the previous version. All of sudden, I no longer have enough space on my root drive for all of my applications. A partitioning system like mine fails when I am unable to accurately predict the amount of space all my apps will require, both now and in the future. And this is becoming increasingly common. A utility that would let me dynamically resize my partitions would be a godsend. Now I don't think the partitioner included in the Boot Camp Assistant does dynamic partition resizing, and it probably never will, and I'll just have to roll with it. But I can dream. And there is another area in which a utility like the one in Boot Camp would have immediate benefits.

I have a client who purchased on of Apple's Xserve RAID systems. This is a pretty cool beast. Fast. Stable. Very reliable. But there's a scalability problem, and it's one reason I won't buy one for my lab. When you read the datasheets and the manuals for the XRAID, you're told that drives can be dynamically added to the RAID in the future, should you ever require more storage. Add the new drive and the RAID will, theoretically, add it to the RAID set and you'll have that much more storage space. This is typical of enterprise-level RAIDs I'm told. It's one of the reasons you get them: scalability. But if you ever try to actually do this on an XRAID you'll run into a little snag. You see, while it is possible to add a drive to the RAID system and have the RAID recognize the new drive and add it to the set, the Mac OS host is responsible for partitioning the RAID. So what happens is, you add the drive, the XRAID sees it and adds it to the set automagically, but the Mac does not recognize the additional capacity. The Mac still sees the original partitioning scheme. I've read through the manuals and discussion forums — hell, I even talked to one of Apple's RAID specialists — and it turns out there is no way to grow the XRAID without repartitioning it from the host Mac. And as you're well aware, I'm sure, there's no way to repartition a Mac drive without wiping it. So, you want to grow your XRAID? Well, you'll have to wipe it. And that just sucks.

Boot Camp on PPC: No Boot Camp for You!
(click for larger view)

Apple's RAID specialist told me that they were working on a solution to this. That was about a year ago, and I've heard nothing since. But the partitioning utility in the Boot Camp Beta gives me great hope that a solution is indeed imminent. It may even be here already, if you have an Intel Mac that is. I've tried it, and no, the Boot Camp Beta will not run on a PPC Mac. No surprise there. But if the information on the Boot Camp page is any indication (Apple calls "Space Maker" the "most elegant hard drive utility ever"), a new and improved partitioning utility is on the horizon for the Mac. And I'm just as excited about this as I am at the prospect of booting Windows on my Mac. Maybe more.

It's been a long time coming.

UPDATE:
I'm told by a colleague that growing a filesystem (like on a RAID) and partitioning a filesystem (what the Boot Camp Assistant does) are two different things, growing being the harder and, according to him, requiring a grow-able filesystem, which HFS+ is not. So maybe this is a pipe dream.

Apple has a FAQ up about the Windows installation process for Intel Macs, and there are some interesting things there. Most notable in the context of this article is the fact that Boot Camp Assistant will only work on "an Intel-based Mac that has one hard disk partition." This does not bode well. Still, I remain hopeful that this utility is indicative of possible future utilities that would allow for more flexible, less destruct ive partitioning.

UPDATE 2:
I've been doing some hunting around on Apple's RAID capabilities and discovered that you can now (as of Tiger, I believe) use Disk Utility to create what called a Concatenated Disk Set (also referred to as JBOD, for Just a Bunch of Old Drives). These sets can be combined with other forms of RAIDs as well. Until yesterday I'd never even heard of Concatenated RAID, but apparently it's the oldest form of RAID. Concatenated RAID doesn't have the redundancy advantages of RAID 0 or the performance advantages of RAID 1. All Concatenated gives you is more contiguous disk space. I've read that Concatenated could be used to append a new drive on an XServe to the RAID set, though all it would do would be to add to the set's capacity and would not be a true member of the set. This is recommended as a stop-gap measure more than anything, and it's recommended that the reformat necessary to truly join the new disk to the set be performed later, at a more convenient time. Interestingly, this seems to suggest that a Concatenated RAID can be performed without wiping the disks in question, though I can't find any documentation that states this one way or the other. Also, the disks in a Concatenated RAIDs do not have to be identical, as with other forms of RAID. Concatenated RAID disks can be all different sizes, shapes and colors. As soon as I can find some drive to offer up as sacrifice, I will be testing out Concatenated RAID. I'll let you know what I find.

UPDATE 3:
It appears that a new verb for the diskutil command has been added in 10.4.6. The verb is resizeVolume and it's what allows the non-destructive partitioning done by the Boot Camp Assistant. Don't look for it in the man pages; it's not mentioned there. But if you just run the diskutil programsan arguments you'll see resizeVolume listed as an option. I first read about this on Daring Fireball, which links to a MacGeekery article that outlines, in-depth, the process of using the new option to non-destructively resize volumes. I don't know yet if this works on PPC Macs — the article suggests it's for Intel machines only. But I'm sure going to give it a whirl. I'll post a follow-up on all my partitioning experiments in the near future. Stay tuned!

External Network Unification Part 1: Research and Development

"Systems Boy! Where are you?"

I realize posting has been lean, lo these past couple weeks. This seems to be a trend in the Mac-blog world. I think some of this has to do with the recent dearth of interesting Mac-related news. In my case, however, it's also due to a shocking lack of time brought on by my latest project.

You may or may not remember, or for that matter even care, about my ongoing Three Platforms, One Server series which deals with my efforts to unify network user authentication across Mac, Windows and Linux systems in my lab. Well, that project is all but done, except for the implementation, which we won't really get to do until the Summer, when students are sparse, time is plentiful, and love is in the air. (Sorry, but if you manage a student lab, you'll probably understand how I might romanticize the Summer months a bit.) Anyway, we've got our master plan for user authentication on our internal network pretty much down, so I've turned my attention to the external network, which is what I've been sweatily working on for the last two weeks.

Our external network (which, for the record, has only recently come under my purview) is made up of a number of servers and web apps to which our users have varying degrees of access. Currently it includes:

  1. A mail server
  2. A web host and file server
  3. A Quicktime Streaming Server
  4. A community site built on the Mambo CMS
  5. An online computer reservations system

In addition to these five systems, additional online resources are being proposed. The problem with the way all this works right now is that, as with our internal network, each of these servers and web apps relies on separate and distinct databases of users for its authentication. This is bad for a number of reasons:

  1. Creating users has to be done on five different systems for each user, which is far more time consuming and error prone than it should be
  2. Users cannot easily change their passwords across all systems
  3. The system is not in any way scalable because adding new web apps means adding new databases, which compounds the above problems
  4. Users often find this Byzantine system confusing and difficult to use, so they use it less and get less out of it

The goal here, obviously, is to unify our user database and thereby greatly simplify the operation, maintenance, usability and scalability of this system. There are a number of roadblocks and issues here that don't exist on the internal network:

  1. There are many more servers to unify
  2. Some of the web apps we use are MySQL/PHP implementations, which is technology I don't currently know well at all
  3. Security is a much bigger concern
  4. There is no one on staff, myself included (although I'm getting there), with a thorough global understanding of how this should be implemented, and these servers, databases and web apps are maintained and operated by many different people on staff, each with a different level of understanding of the problem
  5. All of these systems have been built piecemeal over the years by several different people, many of whom are no longer around, so we also don't completely understand quite how things are working now

All of these issues have led me down the path upon which I currently find myself. First and foremost, an overarching plan was needed. What I've decided on, so far, is this:

  1. The user database should be an LDAP server running some form of BSD, which should be able to host user info for our servers without too much trouble
  2. The web apps can employ whatever database system we want, so long as that system can get user information from LDAP; right now we're still thinking along the lines of MySQL and PHP, but really it doesn't matter as long as it can consult LDAP
  3. Non-user data (i.e. computers or equipment, for instance) can be held in MySQL (or other) databases; our LDAP server need only be responsible for user data

That's the general plan. An LDAP server for hosting user data, and a set of web apps that rely on MySQL (or other) databases for web app-specific data, with the stipulation that these web apps must be able to use LDAP authentication. This, to me, sounds like it should scale quite well: Want to add a new web app? Fine. You can either add to the current MySQL database, or if necessary, build another database, so long as it can get user data from LDAP, as user data is always redundant and should always be consistent. It's important to remember that the real Holy Grail here is the LDAP connection. If we can crack that nut (and we have, to some extent) we're halfway home.

This plan is a good first step toward figuring out what we need to do in order to move forward with this in any kind of meaningful way. As I mentioned, one of the hurdles here is the fact that this whole thing involves a number of different staff members with various talents and skill sets, so I now at least have a clear, if general, map that I can give them, as well as a fairly clear picture in my mind of how this will ultimately be implemented. Coming up with a plan involved talking to a number of people, and trying out a bunch of things. Once I'd gathered enough information about who knew what and how I might best proceed, I started with what I knew, experimenting with a Mac OSX server and some web apps I downloaded from the 'net. But I quickly realized that this wasn't going to cut it. If I'm going to essentially be the manager for this project, it's incumbent upon me to have a much better understanding of the underlying technologies, in particular: MySQL, PHP, Apache and BSD, none of which I'd had any experience with before two weeks ago.

So, to better understand the server technology behind all this, I've gone and built a FreeBSD server. On it I've installed MySQL, PHP and OpenLDAP. I've configured it as a web server running a MySQL database with a PHP-based front-end, a web app called MRBS. It took me a week, but I got it running, and I learned an incredible amount. I have not set up the LDAP database on that machine as yet, however. Learning LDAP will be a project unto itself, I suspect. To speed up the process of better understanding MySQL and PHP (and foregoing learning LDAP for the time being), I also installed MRBS on a Tiger Server with a bunch of LDAP users in the Open Directory database. MRBS is capable of authenticating to LDAP, and there's a lovely article at AFP548 that was immensely helpful getting me started. After much trial and error I was able to get it to work. I now have a web application that keeps data accessed via PHP in a MySQL database, but that gets its user data from the LDAP database on the Tiger Server. I have a working model, and this is invaluable. For one, it gives me something concrete to show the other systems admins, something they can use as a foundation for this project, and a general guide for how things should be set up. For two, it gives us a good idea of how this all works, and something we can learn from and modify our own code with. A sort of Rosetta stone, if you will. And, finally, it proves that this whole undertaking is, indeed, quite possible.

So far, key things I've learned are:

  1. MySQL is a database (well, I knew that, but now I really know it)
  2. PHP is a scripting/programming language that can be used to access d atabases
  3. MySQL is not capable of accessing external authentication databases (like LDAP)
  4. PHP, however, does feature direct calls to LDAP, and can be used to authenticate to LDAP servers
  5. PHP will be the bridge between our MySQL-driven web apps and our LDAP user database

So that is, if you've been wondering, what I've been doing and thinking about and working on for the past two weeks. Whew! It's been a lot of challenging but rewarding work.

This is actually a much bigger, much harder project than our internal network unification. For one, I'm dealing with technologies with which I'm largely unfamiliar and about which I must educate myself. For two, there are concerns — like security in particular — which are much more important to consider on an external network. Thirdly, there are a great many more databases and servers that need to be unified. Fourth, scalability is a huge issue, so the planning must be spot on. And lastly, this is a team effort. I can't do this all myself. So a lot of coordination among a number of our admins is required. In addition to being a big technical challenge for me personally, this is a managerial challenge as well. So far it's going really well, and I'm very lucky to have the support of my superiors as well as excellent co-systems administrators to work with. This project will take some time. But I really think it will ultimately be a worthwhile endeavor that makes life better for our student body, faculty, systems admins and administrative staff alike.