MacFUSE Follow-Up

I recently wondered aloud about the benefits of MacFUSE-supplied ssh access in a web development environment. A reader asked for a follow-up in the comments to that post, so I thought I'd just do a quickie today.

After writing the article, I tried out MacFUSE and the sshfs extension to that bundle. This combo facilitates the mounting of ssh-enabled computers directly on the Mac OS Desktop via a bundled GUI app, foregoing the need for an FTP client application. It was our hope that this would provide a more intuitive and user-friendly experience for our web developers. Unfortunately, in my experience there were bugs: the ssh filesystems often hung or had trouble mounting, often requiring a logout or restart to get things back to normal. (Mind you, this was over a month ago, in Mac OS X 10.4.something-or-other and God-knows-what version of MacFUSE and sshfs.) Also, from a user-simplification standpoint I felt that adding yet another application to the mix (the sshfs GUI app for mounting shares) didn't really do all that much to simplify things. In fact, in some ways, it complicated matters as it is not standard practice anywhere as yet.

That said, I really like the idea of mounting any sort of network share in the Finder. It's what's done in every aspect of our filesharing lives except web development, for some reason (that reason mainly being that Apple, inexplicably, still has not implemented this capability into the Finder.) As I mentioned in the original article, there are styles of development that take place outside the lab that are inherently different from what can happen inside a lab. The use of MAMP, for instance, is not friendly to a shared, multi-user environment, but is perfectly suited to a lone user on a single computer. So, again, I wonder aloud, do we provide a different way of working inside the lab than outside? A completely different workflow?

Ultimately, my answer will mostly be yes, I think. We provide it. We provide it right alongside the sorts of workflows that are common in the industry today. We provide both. We show students. We show teachers. And hopefully, someday, down the line, this will become the standard way of developing for the web. Until then we leave the old ways in place; they are not mutually exclusive.

My concern with MacFUSE and sshfs right now is stability. Because the product requires kernel extensions, and because I saw bugs in my limited testing, I am reluctant to put this on our machines. I can, however, use NFS from within our lab to provide a testing ground for the same sort of behavior that MacFUSE provides (i.e. ssh servers on the Desktop). And this will actually fit in nicely with the way we mount other shares in the lab as well (more nicely than the sshfs GUI, in particular). I will most likely implement this over the summer, however, as there's not enough time in the semester for it to really have much impact. Also, things will probably change a bit with the arrival of Leopard to the lab, and I don't see much point in making folks learn it twice.

Again, external development can be handled differently. So I may recommend, for advanced users, the use of MacFUSE outside the lab if they really end up digging the Desktop web development experience.

That's my take at this point on MacFUSE and web development. Hope it's useful to someone.

Web Programming and MacFUSE

I'm not a web programmer by any means. But a component of the department I work in deals with the web from an artistic standpoint, and a subset of that group does, in fact, do programming. We have web programming classes.

Recently, one of the teachers of one of those classes made the charge that our approach to web programming is old and outmoded. Whether this is true or not is not really the issue. I've been looking for new ways to think about the systems end of that workflow because, well, that's my job, and because it's an inherently interesting challenge to me. How can we make our web development environment more user-friendly?

One general suggestion has been to make the experience more "OS-like." And one step in this direction is to have the web server mount on the Desktop, allowing the developer to work on her site as if it were local. That is, rather than firing up one of the popular SFTP clients and transferring files back and forth from the local machine to the server, the developer could mount her site — or actually, the share her site lives on — directly on the local filesystem. I have two options here: MacFUSE and NFS. I'm testing both currently. So far I've had a couple minor hiccups with MacFUSE's sshfs.app, but it looks to be a fairly smart and user-friendly implementation that web developers here might benefit from. And the NFS approach would work well also, though only from inside the network.

I'm curious what other Lab Admins are doing with regards to web development in their environments. How have you facilitated ease-of-use for a process that's inherently complicated? Or have you? Also, I'm curious if anyone is using MacFUSE — specifically the sshfs component — and what experiences you might have had with it, either positive or negative.

If any of you fine readers have any thoughts on this I would really love to hear them. I've been querying students, staff and faculty for ideas, but haven't come up with much. Maybe things here are perfect, but somehow I doubt it. And, as always, I really just want to make things better.

Please sound off in the comments if you're so inclined.

"Disable Clear Text Passwords" Breaks Randomly

Typically, when you set up a Mac client to bind to a Mac server using Directory Access, there is one lone entry in the Security settings that is checked by default, and that is the "Disable clear text passwords" setting. This seems like a prudent default, and I always leave it checked. I assume that this means that passwords are then sent to the authentication server in some sort of hashed or encrypted form, and that both server and client are set up to negotiate this transaction properly out of the box. Indeed, most of the time this does not present any sort of problem whatsoever.

But for some reason, every now and then, completely randomly, Mac clients will suddenly and mysteriously be unable to authenticate to my Master Authentication Server. Seriously, nothing's changed. Just all of a sudden, Macs can't authenticate. The solution? Un-tick that "Disable clear text passwords" box under the LDAPv3 server configuration's Security tab. Next thing you know, everything's right as rain.


Directory Access: LDAPv3->Select Configuration->Edit
How in Hell Does this Break?

(click image for larger view)

Seriously, can anyone tell me what I'm doing wrong? 'Cause frankly, it's annoying.

External Network Unification Part 5: Almost There

It's been quite some time since I've been able to post anything of any substance. This has a lot to do with the fact that I've been super busy relocating our department and participating in the gut renovation of our lab. This has been an immensely stressful process, but in the end I find that I've learned so much from it, I simply can't complain. I'm coming out a far better SysAdmin than I was going in. And that's a remarkably valuable thing to both me and my employers.

But moving and planning the physical aspects of the new lab has only been a portion of what I've been working on. This renovation has been the perfect opportunity to rebuild our network infrastructure, and part of said rebuilding has resulted in the near completion of our authentication unification project. At this point we've gone from eight different authentication servers — that is, anytime we created a new user, we had to do so on eight different systems — all the way on down to two. Which means that now, anytime we create a new user, we do so on two machines.

Our goal is to get it down to one, hopefully before the Fall semester begins. Our mail server is proving to be the most difficult machine to get working with LDAP authentication, mainly because it authenticates mail users through the wonders of some weird combination of authd, Courier and PAM, and we've yet to crack the magical code that gets these all working in tandem via LDAP. Aside from Mail, though, everything is done. So I thought I'd take a bit of my hard-earned vacation and loosely describe to you how it's all working.

Before I start I'd like to just acknowledge all the help I've had from my fellow SysAdmins in the department. I had a huge amount of assistance on the *NIX server side of things, as well as with network infrastructure and even some last-minute PHP finagling without which this project would have taken significantly longer. In fact, all I really had to do was build the authentication servers and clearly articulate what I wanted. I'm extremely grateful to everyone who helped out.

The little bit of network infrastructure I mentioned is our DMZ. We now have a proper — and more importantly, properly secured — DMZ on which to place an authentication server. I won't go into too much detail here, but suffice to say, having a secure DMZ gives us all kinds of options for authentication between internal and external networks, and makes me feel a whole lot better about using Mac OS X Server as our authentication system for both networks.

Yes, we are using Mac OS X Server to authenticate our entire network. The reason is because Mac OS X Server is the most mature and usable implementation of LDAP for user authentication available on the market today. Is it perfect? No. Is it completely secure? Probably not. Is there anything that even comes remotely close to being able to handle the complexities of user management and database redundancy across platforms with such remarkable ease-of-use? Nope. Nothing. We tried building our own custom LDAP server, which would have been excruciating, and would have taken forever. We tried Red Hat's Directory Server, which looks like it will eventually turn into something to match Mac OS X Server, but which just wasn't yet up to snuff. Nothing matched Mac OS X Server, which did everything we wanted it to, right out of the box and with a minimum of fuss. In fact, once the user database is built, building a Mac OS X master or replica authentication server is a complete and total breeze. At the time of our building and testing it was really the only practical option.

So, here in a nutshell, is what we have:

Internal Network
All authentication originates from the internal network. Passwords can only be changed from the internal network at this time, which is by design. Systems on the internal network include:

  • Master Authentication Server
    Hosts authentication for... Well... Everything, really. This is essentially the same server we used all last year for all our internal authentication needs for Mac, Linux and Windows workstations. It's now being used to push authentication to the external network as well.
  • Internal Replica Authentication Server
    This provides replication of the Master. Should the master fail, the Replica is intended to pick up services (though this doesn't always work perfectly).
  • File Servers
    We have two file servers on the internal network — a Mac and a Linux box — both of which authenticate directly against the Master.
  • Workstations
    We have about 30 Mac, Windows and Linux machines all authenticating to the Master.

DMZ
The DMZ sits between the Big Bad Internet (BBI) and the internal network. It has its own firewall that is fairly strict about what can get in from the BBI. All DMZ authentication originates from the internal network, but is provided by a single server which sits on the DMZ. Systems on the DMZ include:

  • External Authentication Server
    This server is also a replica of our Master, but it's not intended as a failsafe. Rather, it provides authentication services to the entire DMZ. It gets its user database, of course, from the Master. But for other systems to bind to an LDAP server, its role must either be "Master" or "Replica." Setting the role to "Connected to a Directory Server" won't work. In addition to sitting on our DMZ, which is properly firewalled against the harsh realities of the Big Bad Internet (BBI), this system also makes use of its own strict local firewall for an extra added layer of security. Also, all replication communication between Replica (DMZ) and Master (Internal) is encrypted.
  • Data Server
    In addition to unifying authentication, we've also consolidated data storage and access wherever possible. In the past, for instance, movies streamed from the Quicktime Server were stored on that machine's local drive. Web sites were stored on our web server. So, building a web site that used Quicktime Streaming required users to log into two separate machines — the Web Server and the Quicktime Streaming Server. Now we're storing all user-generated content on a separate, dedicated machine — our Data Server — and sharing that machine out to the various servers via NFS. Centralizing this data store means users have only to log on to one server for anything they ever want to do. And also that only that server needs to authenticate users. And yes, that server authenticates them via LDAP on our External Authentication Server. All neat and tidy. Internal and external home account data is still segregated, however — users still have separate internal and external data storage. Though, if we could figure out how to do it securely, this could change.
  • Quicktime Streaming Server
    This machine also uses its own local firewall. It gets its user database from our External Authentication Server over secure channels using the "Connected to a Directory System" as its role currently. Ultimately, however, because of the Data Server, this machine will not need to authenticate users. We are leaving the ability open temporarily to accommodate legacy users.
  • Drupal CMS
    Our new Community site is built on the Drupal engine. We're using the LDAP module to authenticate to the External Authentication Server. Drupal's LDAP module is simple and easy to set up, as is the Drupal system as a whole. So far we're very happy with it.
  • Computer Reservations System
    This is a custom web app built long ago by a former student. We've (and by "we" I mean my colleague) basically hacked the PHP code to authenticatevia LDAP rather than MySQL.
  • Mail Server
    Currently not authenticating to the External Authentication Server. We're working on this and hope to have it working by the beginning of the school year.

The Future
Yes, there's more we want to do. It's always amazing how, once you've completed something, you immediately start seeing ways to make it better.

  • More Redundancy
    Ultimately, in addition to the Replica, I would also like to automate a clone of the Master's boot drive to an external firewire drive as sort of an ultimate safety. Should anything ever go wrong with the Master, I simply plug the firewire clone into virtually any Mac system on the internal network and I'm back on my feet. It might also be wise to have some sort of failsafe for external authentication as well.
  • More Security
    while our setup is fairly secure right now, there are a few areas I'd like to beef up even more when I get a chance. In particular, our CMS connection is not as secure as I'd like it to be. And ultimately I'd like to harden every machine on the DMZ to the best of my ability.
  • More Unification
    Anything else we can unify — and at this point that's mostly internal and external data — I'm open to considering. It's going to be really interesting for me to look critically at what we've done so far and find the flaws and refine the system. But I'll constantly be looking at ways to simplify our current setup even further without compromising security. The easier our network is to use, the more useful it becomes. We've come a long way, but I'm sure we can find even better ways to do things.
  • More Services
    Now that we have an infrastructure in place for user creation, we can add services freely to our network without the worry of creating users for said services. New services need only the ability to authenticate via LDAP. We're already planning an equipment checkout system, and possibly some calendaring systems.

So, I've just finalized the master authentication server. It's done. Built. Finished. Kaput. The rest of our servers are still in various states of finality, and we have until September to lock them down. But right now, unified authentication is, for all intents and purposes (and with the exception of mail), working. And we couldn't be happier. The ultimate test will be, of course, letting users loose on this new infrastructure. I'm betting they'll like it almost as much as we do. At least the ones who know the old system. New users will be none the wiser. Ain't that always the way?

*Sigh*

Replica Reset Voodoo (That Works!)

So today, after downgrading my master server to 10.4.7, I kept getting an error on my replica. So I decided to reset the replica by demoting it to a "Standalone" role, and then re-promoting it to the "Replica" role. But even after doing this, the error message persisted. The message was telling me to check the logs at:

/var/run/openldap-slurp/replica

and doing so did reveal errors like:

ERROR: Type or value exists: modify/add: memberUid: value #0 already exists

The solution was to again demote the replica to standalone status and then archive all the files in:

/var/run/openldap-slurp/replica

to anywhere else. I put them in a folder called "old." Just get 'em out of the way. Once this was done I was able to promote my replica without receiving error messages.

Yay! That wasn't too bad.

Oh, and you may be asking yourself how I knew to do this. Well, to be honest, I don't really remember. I just know that at some point in the past there was a problem I'd had with a replica and it was caused by stale files. So, since my ultimate goal was to start from scratch, I just got everything out of the way. And lo and behold. It worked. Sorry for the voodoo explanation, though. I wish I could be more explicit. Hell, I wish I fully understood what I was dealing with. But I don't. And, though it pains me to say this, I don't have time to figure it out.

But y'know? I'll take the cure even if I don't know what caused the disease.