This is the twelfth in a series revisiting Ball in Your Court columns and posts from the primordial past of e-discovery–updating and critiquing in places, and hopefully restarting a few conversations. As always, your comments are gratefully solicited.
Data Recovery: Lessons from Katrina
[Originally published in Law Technology News, April 2006]
When the sea reclaimed New Orleans and much of the Gulf Coast, hundreds of lawyers saw their computers and networks submerged. Rebuilding law practices entailed Herculean efforts to resurrect critical data stored on the hard drives in sodden machines.
Hard drives operate within such close tolerances that a drop of water or particle of silt that works its way inside can cripple them; yet, drives aren’t sealed mechanisms. Because we use them from the beach to the mountains, drives must equalize air pressure through filtered vents called “breather holes.” Under water, these breather holes are like screen doors on a submarine. When Hurricane Katrina savaged thousand of systems, those with the means and motivation turned to data recovery services for a second chance.
Data recovery, in the words of John Christopher, a veteran data recovery engineer at DriveSavers Inc., (www.drivesavers.com) is “open heart surgery” for hard drives. Companies such as Novato, Calif.-based DriveSavers and Ontrack Data Recovery (a division of Kroll Ontrack Inc., www.ontrack.com) are the courts of last resort for damaged drives. DriveSavers worked on dozens of Katrina-damaged drives, some submerged for weeks. Drive housings were full of crud, and recovery required finding identical drives and sacrificing them for compatible parts. DriveSavers reported that it was able to resurrect data from about two-thirds of the Katrina drives sent in.
Keep Them Wet
Ontrack’s vice president of operations Todd Johnson reports that his company recovered useable data from about 70 percent of the 425 Katrina-damaged drives they received. All the drives required clean room treatment, with the best outcomes seen in those kept immersed in water or sealed in airtight plastic bags until delivery.
“Don’t dry them out,” Johnson warned, because that causes the heads that read data to become affixed to the platters.
Another factor favoring recovery was quick action. Whether you proceed with full-scale data recovery or not, promptly getting a drive cleaned and processed by a professional keeps your options open.
DriveSavers’ Christopher echoed the need to move quickly and resist turning on the power to “see what works.” He lamented that too many dim their prospects for recovery by letting a tech-savvy relative or electronics superstore take a stab at it.
Back It Up and Lock It Down
Despite the miracles performed by professional disk doctors, data recovery is unpredictable and very expensive. Add the cost of business interruption and frustrated clients, and the IT lesson from Katrina is back it up and lock it down. Even when systems survive, they may be inaccessible for prolonged periods due to closed or clogged roadways, hazardous conditions, areas cordoned off to prevent looting or loss of basic services, like electricity and telecommunications. You’ve got to have an accessible back up.
Katrina forced firms across the Gulf Coast to come to grips with flawed backup practices. Many had no backup system at all. Others were horrified to discover that never-tested backup tapes were useless. The proliferation of data on desktop drives and laptops off the backup grid meant that even those diligent about backup suffered data loss. Still others found to their dismay that backups stored in the same city were kept too close.
Whether the risk is hurricane, earthquake, fire, flood, terrorism, theft or disgruntled IT person, no firm is beyond disaster’s reach. Here are steps to help weather the storm:
- Back up critical data…regularly, thoroughly, obsessively.
- Do periodic test restores of backed up data.
- Ensure that key data on laptops and desktops is captured.
- Mass disasters claim entire regions, so store backed up data out of harm’s way. Consider online backup, which safely ensconces data in distant servers, accessible via high-speed net connection from anywhere. Replication in the Cloud is the optimum solution for many; but, remember that what goes up must come down. If you move hundreds of gigabytes or several terabytes into the Cloud, it’s going to take some serious time to download that much data.
- Know the answer to, “What would I grab if I had to leave right now?” Prepare for “grab and go” emergencies by using removable disk drive drawers or external hard drives.
Keep anti-static drive packaging and watertight containers on hand. For desktops, consider a simple and inexpensive SANS or a RAID configurations to make grab and go practical
- Encrypt the back up. Recent high-profile breaches of data security stemmed from poor management of backup media. Be sure your data back- ups are safe from prying eyes and that several in the firm know the encryption key. A backup you can’t decrypt might as well have washed away.
Since I wrote this column, I became sufficiently interested in bringing back dead drives that I sought and obtained formal training and certification as a data recovery specialist; but, my interest was academic–it’s not work I’d want to do for a living. So, when I come across a hard drive that needs open heart surgery, I send it to Scott Moulton at http://www.myharddrivedied.com/. Scott has saved the day for multitudes, and is smart, fun and interesting guy–truly an educator after my own heart in terms of his uncanny ability to explain the most esoteric aspects of digital storage.