A federal court appointed me Special Master, tasked to, in part, search the file slack space of a party’s computers and storage devices.  The assignment prompted me to reconsider the value of this once-important forensic artifact.

Slack space is the area between the end of a stored file and the end of its concluding cluster: the difference between a file’s logical and physical size. It’s wasted space from the standpoint of the computer’s file system, but it has forensic significance by virtue of its potential to hold remnants of data previously stored there.  Slack space is often confused with unallocated clusters or  free space, terms describing areas of a drive not currently used for file storage (i.e., not allocated to a file) but which retain previously stored, deleted files. 

A key distinction between unallocated clusters and slack space is that unallocated clusters can hold the complete contents of a deleted file whereas slack space cannot.  Data recovered (“carved”) from unallocated clusters can be quite large—spanning thousands of clusters—where data recovered from a stored file’s slack space can never be larger than one cluster minus one byte.  Crucially, unallocated clusters often retain a deleted file’s binary header signature serving to identify the file type and reveal the proper way to decode the data, whereas binary header signatures in slack space are typically overwritten.

A little more background in file storage may prove useful before I describe the dwindling value of slack space in forensics.

Electronic storage media are physically subdivided into millions, billions or trillions of sectors of fixed storage capacity.  Historically, disk sectors on electromagnetic hard drives were 512 bytes  in size.  Today, sectors may be much larger (e.g., 4,096 bytes).  A sector is the smallest physical storage unit on a disk drive, but not the smallest accessible storage unit.  That distinction belongs to a larger unit called the cluster, a logical grouping of sectors and the smallest storage unit a computer can read from or write to.  On Windows machines, clusters are 4,096 bytes (4kb) by default for drives up to 16 terabytes.  So, when a computer stores or retrieves data, it must do so in four kilobyte clusters.

File storage entails allocation of enough whole clusters to hold a file.  Thus, a 2kb file will only fill half a 4kb cluster–the balance being slack space.  A 13kb file will tie up four clusters, although just a fraction of the final, fourth cluster is occupied is occupied by the file.  The balance is slack space and it could hold fragments of whatever was stored there before.  Because it’s rare for files to be perfectly divisible by 4 kilobytes and many files stored are tiny, much drive space is lost to slack space.  Using smaller clusters would mean less slack space, but any efficiencies gained would come at the cost of unwieldy file tracking and retrieval.

So, slack space holds forensic artifacts and those artifacts tend to hang around a long time.  Unallocated clusters may be called into service at any time and their legacy content overwritten.  But data lodged in slack space endures until the file allocated to the cluster is deleted–on conventional “spinning” hard drives at any rate.

When I started studying computer forensics in the MS-DOS era, slack space loomed large as a source of forensic intelligence.  Yet, apart from training exercises where something was always hidden in slack, I can’t recall a matter I’ve investigated this century which turned on evidence found in slack space.  The potential is there, so when it makes sense to do it, examiners search slack using unique phrases unlikely to throw off countless false positives.

But how often does it make sense to search slack nowadays?

I’ve lately grappled with that question because it seems to me that the shopworn notions respecting slack space must be re-calibrated.  

Keep in mind that slack space holds just a shard of data with its leading bytes overwritten.  It may be overwritten minimally or overwritten extensively, but some part is obliterated, always.  Too, slack space may hold the remnants of multiple deleted files; that is, as overlapping artifacts: files written, deleted overwritten by new data, deleted again, then overwritten again (just less extensively so).  Slack can be a real mess.

Fifteen years ago, when programs stored text in ASCII (i.e., encoded using the American Standard Code for Information Interchange or simply “plain text”), you could find intelligible snippets in slack space.  But since 2007, when Microsoft changed the format of Office productivity files like Word, PowerPoint and Excel files to Zip-compressed XML formats, there’s been a sea change in how Office applications and other programs store text.  Today, if a forensic examiner looks at a Microsoft Office file as it’s written on the media, the content is compressed.  You won’t see any plain text.  The file’s contents resemble encrypted data.  The “PK” binary header signature identifying it as compressed content is gone, so how will you recognize zipped content?  What’s more, the parts of the Zip file required to decompress the snippet have likely been obliterated, too. How do you decode fragments if you don’t know the file type or the encoding schema?

The best answer I have is you throw common encodings against the slack and hope something matches up with the search terms.  More-and-more, nothing matches, even when what you seek really is in the slack space. Searches fail because the data’s encoded and invisible to the search tool.  I don’t know how searching slack stacks up against the odds of winning the lottery, but a lottery ticket is cheap; a forensic examiner’s time isn’t.

That’s just the software.  Storage hardware has evolved, too.  Drives are routinely encrypted, and some oddball encryption methods make it difficult or impossible to explore the contents of file slack.  The ultimate nail in the coffin for slack space will be solid state storage devices and features, like wear leveling and TRIM that routinely reposition data and promise to relegate slack space and unallocated clusters to the digital dung heap of history.

Taking a fresh look at file slack persuades me that it still belongs in a forensic examiner’s bag of tricks when it can be accomplished programmatically and with little associated cost.  But, before an expert characterizes it as essential or a requesting party offers it as primary justification for an independent forensic examination, I’d urge the parties and the Court to weigh cost versus benefit; that is, to undertake a proportionality analysis in the argot of electronic discovery.  Where searching slack space was once a go-to for forensic examination, it’s an also-ran now. Do it, when it’s an incidental feature of a thoughtfully composed examination protocol; but don’t bet the farm on finding the smoking gun because the old gray mare, she ain’t what she used to be!
See? I never metaphor I didn’t like.

******************************

Postscript: A question came up elsewhere about solid state drive forensics. Here was my reply:

The paradigm-changing issue with SSD forensic analysis versus conventional magnetic hard drives is the relentless movement of data by wear leveling protocols and a fundamentally different data storage mechanism. Solid state cells have a finite life measured in the number of write-rewrite cycles.

To extend their useful life, solid state drives move data around to insure that all cells are written with roughly equal frequency. This is called “wear leveling,” and it works. A consequence of wear leveling is that unallocated cells are constantly being overwritten, so SSDs do not retain deleted data as electromagnetic drives do. Wear leveling (and the requisite remapping of data) is handled by an SSD drive’s onboard electronics and isn’t something users or the operating system control or access.

Another technology, an ATA command called TRIM, is controllable by the operating system and serves to optimize drive performance by disposing of the contents of storage cell groups called “pages” that are no longer in use. Oversimplified, it’s faster to write to an empty memory page than to initiate an erasure first; so, TRIM speeds the write process by clearing contents before they are needed, in contrast to an electromagnetic hard drive which overwrites clusters without need to clear contents beforehand.

The upshot is that resurrecting deleted files by identifying their binary file signatures and “carving” their remnant contents from unallocated clusters isn’t feasible on SSD media. Don’t confuse this with forensically-sound preservation and collection. You can still image a solid state drive, but you’re not going to get unallocated clusters. Too, you won’t be interfacing with the physical media grabbing a bitstream image. Everything is mediated by the drive electronics.

******************************

Dear Reader, Sorry I’ve been remiss in posting here during the COVID crisis. I am healthy, happy and cherishing the peace and quiet of the pause, hunkered down in my circa-1880 double shotgun home in New Orleans, enjoying my own cooking far too much. Thanks to Zoom, I completed my Spring Digital Evidence class at the University of Texas School of Law, so now one day just bubbles into the next, and I’m left wondering, Where did the day go?. Every event where I was scheduled to speak or teach cratered, with no face-to-face events sensibly in sight for 2020. One possible exception: I’ve just joined the faculty of the Tulane School of Law ten minutes upriver for the Fall semester, and plan to be back in Austin teaching in the Spring. But, who knows, right? Man plans and gods laugh.

We of a certain age may all be Zooming and distancing for many months. As one who’s bounced around the world peripatetically for decades, not being constantly on airplanes and in hotels is strange…and stress-relieving. While I miss family, friends and colleagues and mourn the suffering others are enduring, I’ve benefited from the reboot, ticking off household projects and kicking the tires on a less-driven day-to-day. It hasn’t hurt that it’s been the best two months of good weather I’ve ever seen, here or anywhere. The prospect of no world travel this summer–and no break from the soon-to-be balmy Big Easy heat–is disheartening, but small potatoes in the larger scheme of things.

Be well, be safe, be kind to yourself. This, too, shall pass and as my personal theme song says, There's a Great Big Beautiful Tomorrow. Just a Dream Away.

We all need certainty in our lives; we need to trust that two and two is four today and will be tomorrow.  But the more we learn about any subject, the more we’re exposed to the qualifiers and exceptions that belie perfect certainty.  It’s a conundrum for me when someone writes about cryptographic hashing, the magical math that allows an infinite range of numbers to match to a finite complement of digital fingerprints. Trying to simplify matters, well-meaning authors say things about hashing that just aren’t so.  Their mistakes are inconsequential for the most part—what they say is true enough–but it’s also misleading enough to warrant caveats useful in cross-examination.

I’m speaking of the following two assertions:

1. Hash values are unique; i.e., two different files never share a hash value.
2. Hash values are irreversible, i.e., you can’t deduce the original message using its hash value.

Both statements are wrong. Continue reading →

It’s October (already?!?!) and–YIKES–I haven’t posted for two weeks.  I’m tapping away on a primer about e-discovery processing, a topic that’s received scant attention…ever.  One could be forgiven for thinking the legal profession doesn’t care what happens to all that lovely data when it goes off to be processed!  Yet, I know some readers share my passion for ESI and adore delving deeply into the depths of data processing.  So, here are a few paragraphs pulled from my draft addressing the well-worn topic of hashing in e-discovery where I attempt a foolhardy tilt at the competence windmill and seek to explain how hashing works and what those nutty numbers mean.  Be warned, me hearties, there be math ahead!  It’s still a draft, so feel free to push back and all criticism (constructive/destructive/dismissive) warmly welcomed.

My students at the  University of Texas School of Law and the Georgetown E-Discovery Training Academy spend considerable time learning that all ESI is just a bunch of numbers.  They muddle through readings and exercises about Base2 (binary), Base10 (decimal), Base16 (hexadecimal) and Base64; as well as about the difference between single-byte encoding schemes (ASCIII) and double-byte encoding schemes (Unicode).  It may seem like a wonky walk in the weeds; but the time is well spent when the students snap to the crucial connection between numeric encoding and our ability to use math to cull, filter and cluster data.  It’s a necessary precursor to their gaining Proustian “new eyes” for ESI.

Because ESI is just a bunch of numbers, we can use algorithms (mathematical formulas) to distill and compare those numbers.  Every student of electronic discovery learns about cryptographic hash functions and their usefulness as tools to digitally fingerprint files in support of identification, authentication, exclusion and deduplication.  When I teach law students about hashing, I tell them that hash functions are published, standard mathematical algorithms into which we input digital data of arbitrary size and the hash algorithm spits out a bit string (again, just a sequence of numbers) of fixed length called a “hash value.”  Hash values almost exclusively correspond to the digital data fed into the algorithm (termed “the message”) such that the chance of two different messages sharing the same hash value (called a “hash collision”) is exceptionally remote.  But because it’s possible, we can’t say each hash value is truly “unique.”

Using hash algorithms, any volume of data—from the tiniest file to the contents of entire hard drives and beyond—can be almost uniquely expressed as an alphanumeric sequence; in the case of the MD5 hash function, distilled to a value written as 32 hexadecimal characters (0-9 and A-F).  It’s hard to understand until you’ve figured out Base16; but, those 32 characters represent 340 trillion, trillion, trillion different possible values (2¹²⁸ or 16³²). Continue reading →

When computer forensics was in its infancy, examiners collected evidence from disks by copying their contents byte-for-byte to matching, sterilized disks, creating archival and working copies called “clones.”  Cloning drives was inefficient, expensive and error prone compared to the imaging processes that replaced it.  Yet, disk cloning worked for years, and countless cases were made on forensic evidence preserved by cloning and examined on cloned drives.

Now, cloning may be coming back; not to preserve hard drives but  to collect data from mobile devices backed up online, particularly Android phones.  If I’m right, it will be only a stopgap technique; but, it will also be an effective (if not terribly efficient) conduit by which mobile data preserved online can be collected and analyzed in discovery.

Case in point: Google’s recently expanded offering of cheap-and-easy online backup of Android phones, including SMS and MMS messaging, photos, video, contacts, documents, app data and more.  This is a leap forward for all obliged to place a litigation hold on the contents of Android phones — a process heretofore unreasonably expensive and insufficiently scalable for e-discovery workflows.  There just weren’t good ways to facilitate defensible, custodial-directed preservation of Android phone content.  Instead, you had to take phones away from users and have a technical expert image them one-by-one.

Now, it should be feasible to direct custodians to undertake a simple online preservation process for Android phones having many of the same advantages as the preservation methodology I described for iPhones two years ago.  Simple.  Scalable.  Inexpensive.

But unlike the iOS/iTunes methodology, Android backups live in the cloud.  At first, I anticipate there will be no means to download the complete Android backup to a PC for analysis.  Consequently, when we must process the preserved data for litigation, we may need to first restore the data to a factory-initialized “clean” phone as a means to localize the data for collection.  That’s not to say that Google won’t eventually offer a suitable takeout mechanism; after all, Google Takeout capabilities are second to none.  But, until we can backup Android content in a way that it can be faithfully and intelligibly retrieved directly from Google, examiners may revive the tried-and-true cloning of evidence to clean devices then collecting from the restored device.  Everything old is new again.

It won’t be so bad to use this stopgap approach considering that e-discovery typically entails preservation of far more mobile sources than need ultimately be processed.  So, while backing up many online and cloning a few to clean phones certainly isn’t a perfect solution for Android evidence, it’s good enough and cheap enough that courts should give short shrift to parties claiming that preserving phone evidence is unduly burdensome or complex.  For, as my e-discovery colleagues love to say, “Perfect isn’t the standard.”  I agree.  But, neither is the standard, “we couldn’t be bothered, judge.”

I’ve been on something of an e-discovery crusade for the last few years.  No, not my Quixotic, decade-long, “Native Production is More Utile and Efficient” crusade.  This is the other, later-but-just-as-frustrating crusade I call, “Mobile to the Mainstream.”  It’s a relentless, battleship-banging effort to foster recognition that mobile devices and their online information ecosystems are the most important sources of probative electronic evidence we have today.  Unless privileged, mobile evidence should be routinely preserved and produced in mainstream electronic discovery.  Honestly, shouldn’t that be obvious to even the most casual observer of modern life?

That mobile evidence is routinely ignored in civil matters by counsel, government and industry is troubling, and defended–if defended at all–by pointing to the alleged burden and technical “forensic-ness” of marshalling phone content.  I’ve countered with articles showing the ease with which iPhone content can be preserved, extracted and searched–at little to no cost and, crucially, without separating custodians from their devices.  The “trick” for Apple iOS devices was exploiting iTunes, and it was a good trick because iTunes is free, easy to use and supported by Apple on both Mac and Windows platforms around the world.

Then, Apple lately announced it was doing away with iTunes.  ARRRRGGHHH! 😱😖😭

But, no worries, the iPhone backup methodology I’ve put forward is still going to work after Apple releases the new Catalina operating system and cleaves iTunes into dedicated apps for music, podcasts and TV.  In fact, preserving iPhones may be easier for Mac users as Apple is shifting the backup tool into the Finder app.  You’ll do exactly the same thing I wrote about but Mac users with Catalina won’t even need to use iTunes to preserve mobile evidence.  It’ll be built in.

From what I understand, Windows users will still have an app for the task, probably iTunes for the foreseeable future.  So, I’m relieved to know that the “demise” of iTunes won’t be a barrier to simple, scalable preservation of iPhone content.  Things may even get a little easier.