El Keyboardo

I'm still swamped, buried under the avalanche of work that is the direct result of endless delays to the renovation of our department. Nevertheless, I find myself with little to do today. Or, I should say, little I'm actually able to do. Today's delay, from the "I can't catch a break to save my life" department: Keyboards. In Spanish!

Today I finally got a chance to start unboxing and building my new Macs. Opened up the first one and noticed that the keyboard said "Ayuda" where it should have said "Help." Uh... Shit...

So now what? Will I have to send the entire box back for exchange, or just the keyboards? Until I know the answer to this, I can't build my systems. And I won't know 'til Monday, because purchasing is closed this week.

So I blog.

Hola.

"Disable Clear Text Passwords" Breaks Randomly

Typically, when you set up a Mac client to bind to a Mac server using Directory Access, there is one lone entry in the Security settings that is checked by default, and that is the "Disable clear text passwords" setting. This seems like a prudent default, and I always leave it checked. I assume that this means that passwords are then sent to the authentication server in some sort of hashed or encrypted form, and that both server and client are set up to negotiate this transaction properly out of the box. Indeed, most of the time this does not present any sort of problem whatsoever.

But for some reason, every now and then, completely randomly, Mac clients will suddenly and mysteriously be unable to authenticate to my Master Authentication Server. Seriously, nothing's changed. Just all of a sudden, Macs can't authenticate. The solution? Un-tick that "Disable clear text passwords" box under the LDAPv3 server configuration's Security tab. Next thing you know, everything's right as rain.


Directory Access: LDAPv3->Select Configuration->Edit
How in Hell Does this Break?

(click image for larger view)

Seriously, can anyone tell me what I'm doing wrong? 'Cause frankly, it's annoying.

External Network Unification Part 5: Almost There

It's been quite some time since I've been able to post anything of any substance. This has a lot to do with the fact that I've been super busy relocating our department and participating in the gut renovation of our lab. This has been an immensely stressful process, but in the end I find that I've learned so much from it, I simply can't complain. I'm coming out a far better SysAdmin than I was going in. And that's a remarkably valuable thing to both me and my employers.

But moving and planning the physical aspects of the new lab has only been a portion of what I've been working on. This renovation has been the perfect opportunity to rebuild our network infrastructure, and part of said rebuilding has resulted in the near completion of our authentication unification project. At this point we've gone from eight different authentication servers — that is, anytime we created a new user, we had to do so on eight different systems — all the way on down to two. Which means that now, anytime we create a new user, we do so on two machines.

Our goal is to get it down to one, hopefully before the Fall semester begins. Our mail server is proving to be the most difficult machine to get working with LDAP authentication, mainly because it authenticates mail users through the wonders of some weird combination of authd, Courier and PAM, and we've yet to crack the magical code that gets these all working in tandem via LDAP. Aside from Mail, though, everything is done. So I thought I'd take a bit of my hard-earned vacation and loosely describe to you how it's all working.

Before I start I'd like to just acknowledge all the help I've had from my fellow SysAdmins in the department. I had a huge amount of assistance on the *NIX server side of things, as well as with network infrastructure and even some last-minute PHP finagling without which this project would have taken significantly longer. In fact, all I really had to do was build the authentication servers and clearly articulate what I wanted. I'm extremely grateful to everyone who helped out.

The little bit of network infrastructure I mentioned is our DMZ. We now have a proper — and more importantly, properly secured — DMZ on which to place an authentication server. I won't go into too much detail here, but suffice to say, having a secure DMZ gives us all kinds of options for authentication between internal and external networks, and makes me feel a whole lot better about using Mac OS X Server as our authentication system for both networks.

Yes, we are using Mac OS X Server to authenticate our entire network. The reason is because Mac OS X Server is the most mature and usable implementation of LDAP for user authentication available on the market today. Is it perfect? No. Is it completely secure? Probably not. Is there anything that even comes remotely close to being able to handle the complexities of user management and database redundancy across platforms with such remarkable ease-of-use? Nope. Nothing. We tried building our own custom LDAP server, which would have been excruciating, and would have taken forever. We tried Red Hat's Directory Server, which looks like it will eventually turn into something to match Mac OS X Server, but which just wasn't yet up to snuff. Nothing matched Mac OS X Server, which did everything we wanted it to, right out of the box and with a minimum of fuss. In fact, once the user database is built, building a Mac OS X master or replica authentication server is a complete and total breeze. At the time of our building and testing it was really the only practical option.

So, here in a nutshell, is what we have:

Internal Network
All authentication originates from the internal network. Passwords can only be changed from the internal network at this time, which is by design. Systems on the internal network include:

  • Master Authentication Server
    Hosts authentication for... Well... Everything, really. This is essentially the same server we used all last year for all our internal authentication needs for Mac, Linux and Windows workstations. It's now being used to push authentication to the external network as well.
  • Internal Replica Authentication Server
    This provides replication of the Master. Should the master fail, the Replica is intended to pick up services (though this doesn't always work perfectly).
  • File Servers
    We have two file servers on the internal network — a Mac and a Linux box — both of which authenticate directly against the Master.
  • Workstations
    We have about 30 Mac, Windows and Linux machines all authenticating to the Master.

DMZ
The DMZ sits between the Big Bad Internet (BBI) and the internal network. It has its own firewall that is fairly strict about what can get in from the BBI. All DMZ authentication originates from the internal network, but is provided by a single server which sits on the DMZ. Systems on the DMZ include:

  • External Authentication Server
    This server is also a replica of our Master, but it's not intended as a failsafe. Rather, it provides authentication services to the entire DMZ. It gets its user database, of course, from the Master. But for other systems to bind to an LDAP server, its role must either be "Master" or "Replica." Setting the role to "Connected to a Directory Server" won't work. In addition to sitting on our DMZ, which is properly firewalled against the harsh realities of the Big Bad Internet (BBI), this system also makes use of its own strict local firewall for an extra added layer of security. Also, all replication communication between Replica (DMZ) and Master (Internal) is encrypted.
  • Data Server
    In addition to unifying authentication, we've also consolidated data storage and access wherever possible. In the past, for instance, movies streamed from the Quicktime Server were stored on that machine's local drive. Web sites were stored on our web server. So, building a web site that used Quicktime Streaming required users to log into two separate machines — the Web Server and the Quicktime Streaming Server. Now we're storing all user-generated content on a separate, dedicated machine — our Data Server — and sharing that machine out to the various servers via NFS. Centralizing this data store means users have only to log on to one server for anything they ever want to do. And also that only that server needs to authenticate users. And yes, that server authenticates them via LDAP on our External Authentication Server. All neat and tidy. Internal and external home account data is still segregated, however — users still have separate internal and external data storage. Though, if we could figure out how to do it securely, this could change.
  • Quicktime Streaming Server
    This machine also uses its own local firewall. It gets its user database from our External Authentication Server over secure channels using the "Connected to a Directory System" as its role currently. Ultimately, however, because of the Data Server, this machine will not need to authenticate users. We are leaving the ability open temporarily to accommodate legacy users.
  • Drupal CMS
    Our new Community site is built on the Drupal engine. We're using the LDAP module to authenticate to the External Authentication Server. Drupal's LDAP module is simple and easy to set up, as is the Drupal system as a whole. So far we're very happy with it.
  • Computer Reservations System
    This is a custom web app built long ago by a former student. We've (and by "we" I mean my colleague) basically hacked the PHP code to authenticatevia LDAP rather than MySQL.
  • Mail Server
    Currently not authenticating to the External Authentication Server. We're working on this and hope to have it working by the beginning of the school year.

The Future
Yes, there's more we want to do. It's always amazing how, once you've completed something, you immediately start seeing ways to make it better.

  • More Redundancy
    Ultimately, in addition to the Replica, I would also like to automate a clone of the Master's boot drive to an external firewire drive as sort of an ultimate safety. Should anything ever go wrong with the Master, I simply plug the firewire clone into virtually any Mac system on the internal network and I'm back on my feet. It might also be wise to have some sort of failsafe for external authentication as well.
  • More Security
    while our setup is fairly secure right now, there are a few areas I'd like to beef up even more when I get a chance. In particular, our CMS connection is not as secure as I'd like it to be. And ultimately I'd like to harden every machine on the DMZ to the best of my ability.
  • More Unification
    Anything else we can unify — and at this point that's mostly internal and external data — I'm open to considering. It's going to be really interesting for me to look critically at what we've done so far and find the flaws and refine the system. But I'll constantly be looking at ways to simplify our current setup even further without compromising security. The easier our network is to use, the more useful it becomes. We've come a long way, but I'm sure we can find even better ways to do things.
  • More Services
    Now that we have an infrastructure in place for user creation, we can add services freely to our network without the worry of creating users for said services. New services need only the ability to authenticate via LDAP. We're already planning an equipment checkout system, and possibly some calendaring systems.

So, I've just finalized the master authentication server. It's done. Built. Finished. Kaput. The rest of our servers are still in various states of finality, and we have until September to lock them down. But right now, unified authentication is, for all intents and purposes (and with the exception of mail), working. And we couldn't be happier. The ultimate test will be, of course, letting users loose on this new infrastructure. I'm betting they'll like it almost as much as we do. At least the ones who know the old system. New users will be none the wiser. Ain't that always the way?

*Sigh*

Not Dead

Hello out there, oh loyal, faithful readers. All three of you.

Just wanted to pop in and say hi. It's been a solid month since I've posted anything, and that's pretty unusual. I've even stopped responding to comments, which is almost unheard of. Unfortunately it's been somewhat out of my control. Both work and life have been insanely busy. I'm not so much making excuses here as posting this so that anyone who might be reading this site knows that:
A) I am not dead (or even injured)
B) I am not planning on discontinuing the blog.

That said, I wanted to just take a moment and explain what's been going on. Our department is getting a face lift. Which is to say that we are completely remodeling our floor. Gut rehab: Power. A/C. Network cabling. Seriously. Everything is being completely redone. It's freaking fantastic. It's also a hell of a lot of work as I'm the one overseeing the entire transition, from a technological standpoint anyway, which, since this is a digital art program, represents a fairly large piece of the pie. So I'm exceptionally busy helping plan the new floor — from actual per-room computer placement, to the power needs and layout of the server room. Much of this is new to me, so I'm being educated quite rapidly and on the fly. And since this is a gut rehab, we'll have to clear off the floor at the end of the semester, so I'm planning that transition as well — from moving staff computers to another floor and setting them up with an ad-hoc LAN, to moving our entire DMZ to said floor along with the requisite internet pipes. It's not easy, and it's very much a concerted effort. Fortunately, everyone here has really pulled together, and we're actually making it happen somehow. And while I'm elated to finally be getting all this done (it's been a long time coming, believe me, and we desperately need it, if for the cabling and power problems alone — don't even get me started on the A/C!) I'm way too overwhelmed to write about any of it intelligently right now. Or much of anything else for that matter.

We have been forging ahead with our various long term projects, however. Most notably, I've started building and testing an authentication server for the external network. So far it's going exceedingly well, and it shouldn't be long before our external network has a single authentication source (hopefully this summer sometime). I've been taking notes, so expect a post sometime when things get back to normal. We've also recently bought and set up a firewall appliance which we're loving. It's now being used to properly host our internal and DMZ networks, and is really making our network management a thousand times easier and more sensible. We'll be using it for limited VPN as well, which will be sweet. Today a friend will be running a demo of Leopard in his class as well. These — along with all the goodies associated with the renovation — are all posts in the pipe. Just as soon as I get some time.

You know — if I may wax reflective for just a second — it's not even so much that I have no time to write. Clearly, I'm able to write this post. The real problem is that writing about technology takes a certain amount of brain power. No, not even brain power. Brain space is maybe more like it. With everything I'm dealing with, I've got plenty to write about, and even some time to write about it. What I'm missing is the time to actually process what I'm experiencing. And before I can write about it, I have to process it. It's not so much that I need time to write as that I need to time to think. And that I just don't have right now.

So that's what's been going on here. This can't all last forever, of course. And I'm sure I'll be posting regularly again soon. Until then, do check back from time to time. I'll try to post small bits of stuff as I can.

Okay then. Back to the trenches. This has been fun.

Replica Reset Voodoo (That Works!)

So today, after downgrading my master server to 10.4.7, I kept getting an error on my replica. So I decided to reset the replica by demoting it to a "Standalone" role, and then re-promoting it to the "Replica" role. But even after doing this, the error message persisted. The message was telling me to check the logs at:

/var/run/openldap-slurp/replica

and doing so did reveal errors like:

ERROR: Type or value exists: modify/add: memberUid: value #0 already exists

The solution was to again demote the replica to standalone status and then archive all the files in:

/var/run/openldap-slurp/replica

to anywhere else. I put them in a folder called "old." Just get 'em out of the way. Once this was done I was able to promote my replica without receiving error messages.

Yay! That wasn't too bad.

Oh, and you may be asking yourself how I knew to do this. Well, to be honest, I don't really remember. I just know that at some point in the past there was a problem I'd had with a replica and it was caused by stale files. So, since my ultimate goal was to start from scratch, I just got everything out of the way. And lo and behold. It worked. Sorry for the voodoo explanation, though. I wish I could be more explicit. Hell, I wish I fully understood what I was dealing with. But I don't. And, though it pains me to say this, I don't have time to figure it out.

But y'know? I'll take the cure even if I don't know what caused the disease.