So we had yet another calamity in the Systems Boy household last week: A hard drive failure in a four year old, 15" PowerBook. Oddly, a workmate had the exact same thing happen to him within days of our catastrophe. In fact, there's been all manner of hardware failure in recent days. I know that drives are prone to dying after a number of years, but geez! It sure seems like lately there's been a steady shit stream aimed squarely at the tech fan. Makes me ponder the more cosmic aspect of this biz.
[Gazes dreamily off into space for a moment. Then abruptly snaps to.]
The trigger for this failure, ironically, was our attempt to make a backup. (Oh, technology gods, thou art a riot!) See, our original goal was to update the OS to Leopard, but with all the craziness going on these days we decided to clone the drive before we proceeded with said update. But in the course of cloning, it would appear in retrospect, we hit a bad block and triggered the first of what would be many, many disk errors. Unable to pull a backup, we began our descent into drive repair hell in our latest heroic attempt to salvage that ever-important thing contained on and lost from drives: the data.
File-Level Attempts
Our first try was with Disk Utility, which consistently reported, in all red text, that it could neither verify nor repair the file system. Right. On to attempt number two.
Disk Warrior is my go-to utility for any sort of file system damage that Disk Utility is unable to repair. I've rarely seen a disk that one of these two apps couldn't fix. Today would be one of those rare days. After mounting the drive on a known good system using Target Disk Mode, we let Disk Warrior perform its initial scan of the drive. What we found was decidedly ugly. Disk Warrior told us that it was unable to replace the borked directory with its shiny new, replacement directory because of a "disk malfunction."
That's when we knew the drive was fried.
When a hard drive has problems, 99% of the time those problems are directory related. That is, the hard drive contains data about the files on disk — where they belong, how many there are, how the disk is partitioned and so on. And usually, when there is a problem with a drive, it is because this information has been corrupted somehow. These days there are numerous utilities that can easily and accurately repair these sorts of problems, Apple's included Disk Utility among them. Sometimes the damage is too extensive, though, so we turn to something a bit more drastic, like Disk Warrior. Disk Warrior forgoes the repair, and instead scans the disk and creates a brand-spankin' new directory, replacing the broken one with its new one once you've made sure everything is cool, and perhaps made a backup. Now, when Disk Warrior is unable to do this it's indicative of a much more serious problem. When this happens it is very likely that the drive hardware is beginning to fail.
Time for a new drive.
What Disk Warrior does in these instances is it shows you the best picture it can muster of the drive's contents in a read-only preview, and then advises you to backup as much as you can before total failure. So that's what we did. You're never sure how much time you have in these situations, so we went through folder by folder trying to locate and backup the most important files first. With each successive copy the drive became slower and slower. Luckily, we were able to pull the most recent, most important files. Most everything else was backed up or able to be easily reconstructed.
Block-Level Attempts
Once we had gotten the most important stuff we decided to see what else we could get. I tried running some rsync commands and got some stuff that way, but not much, and it was taking forever. Once I'd given up trying things at the file level, I decided to make my last ditch effort with a well-worn but powerful little UNIX command called, simply, dd. (No, it does not stand for "Drives Die," though maybe it should.)
The dd command reads data from a disk at the block level and copies it from standard input to standard output which can then be written to a file of your choosing. I use dd by running it on the /dev entry of the drive in question and writing the output to a disk image file (DMG):
sudo dd bs=512 if=/dev/disk3s3 of=/Volumes/Work/LastDitch-DD-01.dmg conv=noerror,syncThe good thing about dd is that you can instruct it to skip damaged sections of the disk. That's what the "noerror" option is for. The downside to dd is that it wants to read the entire disk, and that makes it very slow. In this instance I was not able to rescue any data, mainly because, as I soon discovered from my dd runs, the disk was just too far gone. I did learn some interesting strategies for using dd to recover data though.
The first thing you can try if dd is running slowly is to increase the block size. This is how much data dd will consider before moving to the next read. The default is 512 bytes. I've read upping that to 51200 will sometimes yield speedier results:
sudo dd bs=51200 if=/dev/disk3s3 of=/Volumes/Work/LastDitch-DD-02.dmg conv=noerror,syncIn my case it did not, primarily, I believe, because there was a problem in the beginning of the drive, and dd was having trouble moving past that spot. So another thing you can tell dd to do is to skip a certain portion of the drive, say the first 2 GBs:
sudo dd bs=51200 if=/dev/disk3s3 skip=2000000 of=/Volumes/Work/LastDitch-DD-03.dmg conv=noerror,syncFinally, you can also tell dd to only write in 1 GB chunks, using the count option:
sudo dd bs=51200 if=/dev/disk3s3 count=1000000 skip=2000000 of=/Volumes/Work/LastDitch-DD-03.dmg conv=noerror,syncI was getting some good results after having skipped the first 2 GBs — apparently they were really damaged — so I decided to write a script that would skip the first 2 GBs and then begin writing out 1 GB chunks of data. It would've looked something like this:
sudo dd bs=51200 if=/dev/disk3s3 count=1000000 skip=2000000 of=/Volumes/Work/LastDitch-DD-Chunck-01.dmg conv=noerror,sync
sudo dd bs=51200 if=/dev/disk3s3 count=1000000 skip=3000000 of=/Volumes/Work/LastDitch-DD-Chunck-02.dmg conv=noerror,sync
sudo dd bs=51200 if=/dev/disk3s3 count=1000000 skip=4000000 of=/Volumes/Work/LastDitch-DD-Chunck-03.dmg conv=noerror,sync
...Etc, etc, up to the 40 GBs needed to scour the drive. I never got to write the script, though, because the last dd command seized up and the drive began making the clicking, knocking and whirring sounds of its agonized and tortured death. It was quickly dying. We could do no more.
At this point, mainly for my own edification, I decided to see what could be done outside the confines of my home office. I decided to get a quote from Drive Savers.
Hardware-Level Attempt
Drive Savers, perhaps wisely, does not list prices for their services on their website. To get an estimate you have to give them a call. When I called them I was greeted by a very friendly and helpful service person — yes, person — which was really nice. The last thing you want to deal with when you're having a mechanical failure is a machine. The person on the other end of the line asked me a few basic questions to gauge what state the drive was currently in, things like what attempts I had made to rescue the data, would the drive mount, and the like. After entering this info into her systems she directed me the "Tips, Techniques and Solutions" page on their website (very useful — love the drive sound audio samples), stressing above all that in order to have the best chance of recovery at this point the drive should not be powered on again. She also offered up some information about the company and what they do: For one, they started with Mac data recovery and are an all-Mac shop, which surprised me a little. She also pointed me to information on the Drive Savers clean room, a vital part of data recovery at the hardware level. She then took my email and contact info and gave me both a written and verbal estimate of how much I could expect to spend should I decide to go ahead and have Drive Savers attempt to save my data (I don't think they'll actually save the drive). All in all it was a very pleasant and informative experience. Normally I am loathe to use the phone for business, but Drive Savers really seems to know what they're doing, at least when it comes to pre-sales customer service, and that counts for a lot in my book.
This is, of course, all prep for the fact that, if you do want to make the attempt at data recovery, you'll be expected to drop a significant amount of money. This is hardly surprising. Those clean rooms don't look particularly cheap to build or maintain. And if data recovery at the hardware level is anything like it is at the software level, it is a laborious and time consuming process. I was given a range of prices ($500-$2700 dollars) and told that the cheapest I could expect to get away with — the economy plan, which isn't as fast as some of the other, more expensive plans — was $500 dollars. But it was likely I'd pay somewhere closer to the upper third of the range, more like $1500 to $2000 dollars. It all depended, of course, on how much data Drive Savers could recover.
I didn't really find these prices particularly surprising. I'd long heard how much such a recovery could cost. That it would be pricey. I was glad that I was not in a situation that required me to fork out this amount of money. I'm glad such a service exists for the odd catastrophe, though I hope never to have to use it. Drive Savers' website offers advice on keeping backups:
"Backup strategies:
* Invest in redundant backup systems
* Establish a structured backup procedure to make copies of all critical data files, using software compatible with the operating system and applications
* Periodically test the backups to verify that data, especially databases and other critical files, are being backed up properly
* Keep at least one verified copy of critical data offsite"
Sage advice, all. Take it from those who know all too well.
The Belly of the Beast
Once we'd decided not to use a hardware data recovery service the only thing left to do was spec out, buy and install a new hard drive. This wasn't terribly difficult, but as is so often the case, there was the odd snag or two.
Before we even bought a drive, I wanted to see how hard it would be to open the PowerBook for servicing. If it was going to be a bear — and some PowerBooks are certainly easier to crack than others — I'd let the fine technicians at Tekserve do the job. So I went in search of manuals and instructions for this particular model of PowerBook. Without too much trouble I was able to locate, at Apple's site, the manual for our 1.67 MHz, 15" Aluminum PowerBook. It contained no instructions for hard drive replacement, which is generally a sign that Apple would rather you not attempt the repair yourself. That got me a little worried.
Finally, however, I found instructions — great instructions, no less — at the venerable — awesome, actually iFixit.com. iFixit, for those of you who don't know, provides step-by-step, illustrated guides on taking apart and performing repairs on Apple hardware. For free. They're amazing. I feel guilty not buying anything from their site. Oh yeah, they also sell parts, tools and service as well. I love them. And from what I could see, the repair would be tedious — lots of screws — and would require a trip to the hardware store — blasted tiny hex screws! — but it would be doable. Still, taking things one step at a time, I thought I'd perform the teardown before buying the drive. Just in case.
And perform I did. Using iFixit's excellent guide, I was able to crack the PowerBook in short order. I was ready to buy a drive.
Buying a Drive
There are two things SysAdmins typically are, particularly when it comes to technology: cheap and lazy. Hunting for a replacement drive brought both of these qualities in my personality to bear. I was looking for the cheapest replacement I could find, at the location closest to my house, a SysAdmin's dream hunt. The closest proper computer tech shop to me is Tekserve, with Best Buy a close second. Tekserve doesn't list what bare drives they carry, if any. But Best Buy seems to have the goods. But Best Buy is still a good half hour train ride, so I did some physical recon at my nearest Radio Shack, which happens to be right around the corner. They informed me that, though they did not have any bare drives in stock, they did have portable USB drive on sale. Drives from which I could pull and the internal component and install it in the now drive-less PowerBook. In fact, they had a 160 GB Iomega Prestige for less than a bare drive would have run me at Best Buy — a mere $75 clams post-sales-tax. Not bad. I took it.
I'd like to pause here and see if anyone can guess why this didn't work out for me. You have pretty much all the data you need in this article to figure it out. But don't feel bad if you can't. The good lord knows I surely didn't. I'll wait a minute... Pretend there's Jeopardy countdown music playing... Aaand...
Okay. Did you guess it?
I got the drive home, popped it out of its case and went to put it in the open PowerBook. But it didn't fit. (Have you guessed it yet?) Here's the thing: PowerBooks use 2.5" ATA drives (Parallel ATA, or PATA), but drives in today's externals are all now SATA (Serial ATA) drives. Blast!
Oh well. At least it was cheap.
Another quick look at the web revealed that all the bare drives at Best Buy were SATA as well. Blast again!
The nearest ATA drive I could find was at J&R, which is all the way downtown, almost at the very tippy-tip of Manhattan — far. So that's where we went.
Once we got back, we installed the drive and — the very first thing to go right all day — it worked. Perfectly. Things were finally looking up.
Once we had installed the drive it was simply a matter of formatting it, installing the latest version of Leopard (which is all we ever wanted to do in the first place) and copying over the rescued and reconstructed data. Oh, did I mention that the reason the client wanted Leopard was for Time Machine? Yup. Backups. Great timing. So we set up Time Machine as well. All that went exceedingly smoothly and our repair is, at last, complete. Whew! What an ordeal!
But, man, did I ever learn a lot.
The Life and Death of Hard Drives
So yes, drives die. How they die, though, is almost as important as how they lived, and certainly as interesting. It's somewhat comforting to know that this drive, while quite dead indeed, did not die in vain. Rarely have I had the opportunity to learn so much about practical drive recovery. I have that PowerBook drive — specifically its death, in fact — to thank for my lesson.