A Good Time to Leave Wordpress

Speaking of leaving Wordpress, it looks like the blogging platform is under a major attack. Wordpress has long been known for its security holes. Not that they're necessarily any worse than similar platforms, but the fact that Wordpress is both so prevalent and so insecure makes it an obvious target. 

Huge attack on WordPress sites could spawn never-before-seen super botnet
Security analysts have detected an ongoing attack that uses a huge number of computers from across the Internet to commandeer servers that run the WordPress blogging application.

I was always pretty good about keeping my Wordpress installations updated with the latest security patches, which is part of what made it such a bear to manage. Glad I don't have to mess with that anymore.

Talk about good timing!

The End Of IT?

It's rare that I see a blog post with comments that are consistently smarter and more well-informed than the post itself. But, folks, we have a winner. This article by 37Signals' "David" is just such a mythical beast. It's so infuriatingly bad, so completely misinformed, and so utterly borne of ignorance and frustration, that I think I'll just go through bit by bit and explain why people should just stop posting this utter nonsense. (And, Gruber, shame on you for thinking there was anything even resembling a well-reasoned argument here.)

David begins his rousing critique of the IT industry thusly:

"When people talk about their IT departments, they always talk about the things they’re not allowed to do, the applications they can’t run, and the long time it takes to get anything done. Rigid and inflexible policies that fill the air with animosity. Not to mention the frustrations of speaking different languages. None of this is a good foundation for a sustainable relationship."

True enough. This is often what people talk about when they talk about IT. They rarely talk about how awesome it is that they have a usable network or rooms full of computers without viruses. But let's continue.

"If businesses had as many gripes with an external vendor, that vendor would’ve been dropped long ago. But IT departments have endured as a necessary evil. I think those days are coming to an end."

Typically, businesses don't have gripes with IT, end-users do. But, okay, I'm curious to hear your reasoning.

"The problem with IT departments seems to be that they’re set up as a forced internal vendor. From the start, they have a monopoly on the 'computer problem' – such monopolies have a tendency to produce the customer service you’d expect from the US Postal Service. The IT department has all the power, they’re not going anywhere (at least not in the short term), and their customers are seen as mindless peons. There’s no feedback loop for improvement."

I don't think that that's really the problem with IT departments at all. The problem is that many IT departments make crappy policy decisions that are user-hostile. But that's not because they have "all the power." In fact those decisions are often, I'd suspect, borne out of a need to satisfy certain technical goals using limited resources. The characterization that IT departments see their customers "as mindless peons" is offensive to anyone who works in this business, and generalizations such as these do as much to "fill the air with animosity" as any IT policy does. Clearly, the flip-side of "the problem" is an almost willful ignorance on the part of certain members of the tech biz — David, I'm looking at you — to make even the slightest effort to understand what IT departments do before making grand proclamations on the internet about the "end of IT." While I do agree that there should be better avenues for feedback, that doesn't mean I can always get what I want. And crying about it is a five-year-old's tactic.

"Obviously, I can see the other side of the fence as well. IT departments are usually treated as a cost center, just above mail delivery and food service in the corporate pecking order, and never win anything when shit just works, but face the wrath of everyone when THE EXCHANGE SERVER IS DOWN!!!!!"

You're goddamned right about that. I suspect that a well-respected, well-treated IT department would have warmer, fuzzier feelings for its "customers." But the fact is that, because people like David continue to see IT departments simply as "cost centers" and not as members of a single team with a shared goal, IT departments continue to be reviled, often by members of the very corporate structures upon which they depend. Unfortunately, this relationship has been sustainable for over twenty years. Probably because, in many institutions, it is a relationship that, though pathalogical in many ways, is necessary.

"At the same time, IT job security is often dependent on making things hard, slow, and complex. If the Exchange Server didn’t require two people to babysit it at all times, that would mean two friends out of work. Of course using hosted Gmail is a bad idea! It’s the same forces and mechanics that slowly turned unions from a force of progress (proper working conditions for all!) to a force of stagnation (only Jack can move the conference chairs, Joe is the only guy who can fix the microphone)."

No, IT job security is not "dependent on making things hard, slow, and complex." I'm so tired of hearing that. It's simply not true, and I'd love to hear a concrete, real-world example of some place where that was the case. The fact of the matter is, IT job security is dependent on making things work. Period. If you really think that the IT department uses Exchange Server so that their buddies can get a job, you simply don't have a clue what IT does.

"But change is coming. Dealing with technology has gone from something only for the techy geeks to something more mainstream. Younger generations get it. Computer savvyness is no longer just for the geek squad."

Change may be coming. Indeed, I hope it is, because I would love to see the relationship between IT and the end-user improved upon, and, where possible, lessened or even ended. And certainly "dealing with technology" is something everyone has to do these days, but after working in tech education for eleven years, I see no evidence that people have gotten any tech-saavier at all. In fact, from one year to the next, people seem to be pretty much the same: they're either tech-saavy or they're not. It has less to do with exposure, more to do with personality. Some people can sing, some people can't. Making the bad singers listen to music all day doesn't make them good singers.

"You no longer need a tech person at the office to man 'the server room.' Responsibility for keeping the servers running has shifted away from the centralized IT department. Today you can get just about all the services that previously required local expertise from a web site somewhere."

Apparently, David, you seem to think that all IT does is run servers, which you seem to think requires them to stand next to them inside a server closet somewhere. Hate to break it to you, buddy, but IT does way more than run your shitty-ass fucking servers. IT configures your switches; they deploy your workstations to your labs; they build and maintain your render clusters, your RAIDs your SANs; they provide all your network infrastructure and keep your workstations virus- and botnet-free. And they usually do it from some sunless underground cavern because idiots like you fail to see their importance. You cannot get any of those things from a website.

"The transition won’t happen over night, but it’s long since begun. The companies who feel they can do without an official IT department are growing in number and size. It’s entirely possible to run a 20-man office without ever even considering the need for a computer called “server” somewhere."

Again, your obsession with servers. And again, I'd love to see some numbers on this. But okay, let's assume for a minute that you're right. What you're basically saying is that there are a lot more smaller companies forming on a regular basis out there. And, sure, smaller companies don't need an IT department. But smaller companies never needed an IT department. Smaller companies could always outsource their technology needs. That's not new. That's not change. That's just more proof that you don't know the first thing about what IT departments are or what they do.

"The good news for IT department operators is that they’re not exactly saddled with skills that can’t be used elsewhere. Most auto workers and textile makers would surely envy their impending doom and ask for a swap."

And that's proof that you're a condescending asshole.

Finally, for the straight shit on what IT actually does, John C. Welch says far more than I ever could (as he actually works in IT) and with far fouler language.

UPDATE:

Here's the inimitable Mr. Welch's response to the very same article. See? I told ya so. (Thanks, John Mahlman, for the tip.)

Archives: Redux

My recent Archives article was met with some controversy and debate, which is great. I love controversy and debate, and a terrific discussion ensued. That discussion has led me to think a bit harder on my archive plan, and I'd like to follow up on the matter with some of the specifics of said plan, and expand on some of the ideas touched on therein.

It's Personal

In the Archives post I basically said I'd be archiving all my "non-essential data" to hard drives and reserving optical media archives for only the most essential archives. I should first point out that what I am talking about here is my personal data. This is not necessarily a method I'd use at work or for a client. Archive methods should be specific to the needs of the situation.

The Future

One of my rationales for using hard drives was that hard drives are more likely than optical to be accessible in 10 years with the equipment of the day. It's this particular idea that received a great deal of criticism, and I'm starting to see why.

Just a few weeks ago I had occasion to archive some museum kiosks that ran from some very old PowerMacs. Luckily, these PowerMacs were just barely of the era when ATA drives were starting to be used as internal drives on Macs. Getting the data off these systems was fairly straightforward. I simply hooked PowerMacs' the ATA drives up to a firewire case and archived the data to DMG. Shortly thereafter, however, I wanted to perform a similar process with a slightly earlier vintage PowerMac. This machine, however, contained a SCSI drive. And finding a way to access and archive this drive proved almost impossible without going to extreme lengths and making obscure hardware purchases. Had there been some kind of optical archive of these systems, I would have almost certainly been able to pull a backup using today's equipment.

I'm not sure what the future of optical media is. Until recently, I was pretty convinced it was not long for this world and would surely be displaced as a distribution medium by the web. But after thinking on the comments to that article, and talking to people way smarter than me on such matters, I realize I may be wrong. And if that's the case, optical will be more likely to be readable than hard drives ten years in the future. But whatever the case, this is certainly true for media from ten years ago. You're more likely to be able to read ten year old optical media than you are hard drives of that era.

Non-Essential Data

That said, I'd like to clarify the "non-essential data" qualifier I tossed in in the article. To be clear, I'm not completely eschewing optical media for my archives. What the article represented was my shift from optical as my only form of backup to hard drives as a significant if not primary form of data backup and archive.

To get even more specific, in the past I archived everything to optical media. But with the huge amounts of data I now collect, that's not really so practical anymore, nor is it necessary. So these days the bulk of my data — large, non-essential data, things like ripped DVDs, video captures from tape, software installers, and data with a shelf life (i.e. that is only useful for a period of time or that relies on old versions of software or hardware) etc. — will be archived to hard drive. This will allow easy storage and retrieval. And it should last long enough. The idea is that this data isn't forever data. It's stuff I want to keep around for a while, but if I haven't needed it in ten years, I probably won't ever need it again.

More important data — of which there's really not that much, but stuff like big video projects (sans captured media), photos, my websites, contacts, stuff that would really kill me to lose — I'll be burning to optical. That way I have double backups of it (I'll also keep it in the hard drive archive), and I'll have it on a more robust medium that may have a better chance of being readable than hard drives in the future.

So what's really going on here, for me, is a prioritization of my data backups that's reflected in my archive procedures. With this prioritization, I can now rely much more heavily on hard drives as an archive medium. Using hard drives I can back up and access a lot more stuff with much greater ease and speed. Doing this allows me to use optical media only for the most important data. But make no mistake: optical will still be an important component in my backup strategy.

Live Archive

I wanted to also take a minute to mention one way hard drives are somewhat future-proof and useful as a true archive, and this is the idea of a live, rolling archive.

In the lab where I used to work we kept — or tried  to keep — a long-term archive of all student work that was accessible to incoming students so that they could look at and benefit from the work of their predecessors. Our students made all sorts of work, from web projects to video and animation projects to installations. And their work was initially being archived to all manner of media, from tape media to optical. There was no standard. By the time I got involved there were projects going back ten or fifteen years, and it was becoming clear that, no matter what medium we used today, we'd need to re-archive everything every so often as data access techniques and hardware evolved. I believe that, in a case like this, where the archive is constantly growing and reaches back well over ten years, but to which access is always required, the concept of the hard-drive-as-archive-medium is a sound one. The implementation would be fairly simple in concept: everything — the entire archive — is kept on a hard drive to which the community has access. As the archive grows, say every few years, it is transferred to larger storage. As storage standards change, it is transferred to the latest greatest medium of the day. Of course, redundant backups are also kept of the entire archive. But since this data is constantly being re-archived, hard drives — or whatever replaces them in the future — make for a sensible way to have a rolling, live archive, and reduce the need for more permanent solutions like optical. Perhaps Chucky, in the comments to Archives, put it best:

"In other words, hard drive archival demands cycling your backups over time to new hard drives with fresh magnetic media and evolving HD interfaces."

I guess the overarching lesson here, if there is one, is that your archive method should reflect the specifics of your situation; there is no one archive method for everyone. The corollary to that, for me, is that hard drives can (and will) now be a significant part of my archive method.

The Real Cloud

When I first saw the title of this article over at Ars I rolled my eyes: Cloud Computing: a short introduction

Then I read it.

It turns out to be an incredibly thorough, yet brief, technical description of the term Cloud Computing. The article does a great job of defining Grid Computing and then uses that definition as point of comparison to arrive at a very real and very useful definition of The Cloud. I'll be a little less embarrassed to use the term after reading it. Only a little. But still...

Well worth a read for anyone who needs some clarity on the whole Cloud thing.

Why We Tell You To Reboot: Redux

I recently wrote an article, entitled Why We Tell You To Reboot, that described a Final Cut Pro bug which, after going to great troubleshooting lengths, I was ultimately only able to solve by a simple reboot. Shortly after posting I received a single comment from a fellow admin and blogger:

"You really tell people to reboot for no particular reason?

I don’t believe we should accept that standard from OS X, and what kind of an IT person are you if you’re blindly proposing solutions without any reasoning to back them up?"

What kind, indeed.

I posted a response to the comment that basically explained my position in a nutshell, but I thought it was worth writing a follow-up on the question, both for thoroughness and for those who didn't happen by the comments section of the previous article, or who may have had a similar reaction.

Reboot!

When a user comes to me with a problem, my primary goal is to fix the problem and get the user working again. Typically what happens is that we have a little discussion about what's going on. Once I feel I have a good handle on the symptoms, often the next words out of my mouth are, "Did you reboot?" If the answer is no, and the situation permits, I will recommend that they do so. Rebooting is often my first step in troubleshooting.

I believe (though I can't be completely sure) that my commenter took issue with the approach because blindly telling the user to reboot to solve problems gives the sysadmin no information about what those problems are and what caused them. But in fact, as I'll demonstrate in a moment, it's not blind, and it does tell us one important thing: rebooting either solves or doesn't solve the problem. This in and of itself can be crucial troubleshooting information if there is a deeper problem at work.

But the fact of the matter is that, probably 80% of the time, there is no deeper issue. The fact is that rebooting routinely fixes problems with no other practical solution, such as the one I described in my article. Moreover, it provides the end-user with a method of troubleshooting that is likely to achieve the desired results — allowing them to get the system or an application back to a working condition — without the need for admin intervention. This is win-win: it saves both the user and me time and energy and, by determining whether or not a reboot is helpful, still provides valuable troubleshooting information.

I would even argue that rebooting should almost always be the first step in troubleshooting. When a user comes to me with a problem, I have no idea what they've been doing on that system. I have no idea how many nor which applications they have open. I have no idea what sorts of preferences they've set. There's simply no way for me to reliably predict the state the user has put their machine in, and thus whether or not this is a system- or user-level problem. The only way for me to get things back to some semblance of a known, working state is to reboot the system. Rebooting has myriad benefits, not the least of which are: clearing stale caches; recreating network connections; and freeing up RAM and disk space. In fact, it seems almost crazy to proceed with most troubleshooting without first rebooting.

You may have noticed that I keep saying that rebooting should almost always be the first troubleshooting step. That almost is there because, obviously, there are times when rebooting is not a good first step. Primarily, when a user stands to lose work by rebooting. If an application is hung and the user hasn't saved his document, for instance, I don't tell them to reboot. Rebooting would be bad in this instance. Also, I usually try other troubleshooting methods first on my own systems, with which I am better acquainted (though this is often to my own detriment and rebooting would have been quicker and easier, as happened with the Final Cut bug).  Another instance in which I avoid a reboot is when there is a persistent problem that is not solved, or is only temporarily solved, by a reboot. Then I do need to get on that system and attempt to understand a problem. And that's what I do.

But, because Mac OS X is very reliable on the whole, these instances are extremely rare in my experience. The majority of problems are minor and are easily and permanently rectified by a simple reboot. I stand behind that recommendation, and any search of Mac troubleshooting articles will reveal that the advice is almost universal. That's because it works.

So, hopefully it's clear by now that I'm not "blindly proposing solutions without any reasoning to back them up." Hopefully it's clear now that there are a lot of good reasons to try rebooting as a first troubleshooting step.

And hopefully it's clear that the kind of sysadmin I am is the kind that likes to get his users back up and running again with a minimum of friction.