We all need certainty in our lives; we need to trust that two and two is four today and will be tomorrow.  But the more we learn about any subject, the more we’re exposed to the qualifiers and exceptions that belie perfect certainty.  It’s a conundrum for me when someone writes about cryptographic hashing, the magical math that allows an infinite range of numbers to match to a finite complement of digital fingerprints. Trying to simplify matters, well-meaning authors say things about hashing that just aren’t so.  Their mistakes are inconsequential for the most part—what they say is true enough–but it’s also misleading enough to warrant caveats useful in cross-examination.

I’m speaking of the following two assertions:

1. Hash values are unique; i.e., two different files never share a hash value.
2. Hash values are irreversible, i.e., you can’t deduce the original message using its hash value.

Both statements are wrong. Continue reading →

It’s October (already?!?!) and–YIKES–I haven’t posted for two weeks.  I’m tapping away on a primer about e-discovery processing, a topic that’s received scant attention…ever.  One could be forgiven for thinking the legal profession doesn’t care what happens to all that lovely data when it goes off to be processed!  Yet, I know some readers share my passion for ESI and adore delving deeply into the depths of data processing.  So, here are a few paragraphs pulled from my draft addressing the well-worn topic of hashing in e-discovery where I attempt a foolhardy tilt at the competence windmill and seek to explain how hashing works and what those nutty numbers mean.  Be warned, me hearties, there be math ahead!  It’s still a draft, so feel free to push back and all criticism (constructive/destructive/dismissive) warmly welcomed.

My students at the  University of Texas School of Law and the Georgetown E-Discovery Training Academy spend considerable time learning that all ESI is just a bunch of numbers.  They muddle through readings and exercises about Base2 (binary), Base10 (decimal), Base16 (hexadecimal) and Base64; as well as about the difference between single-byte encoding schemes (ASCIII) and double-byte encoding schemes (Unicode).  It may seem like a wonky walk in the weeds; but the time is well spent when the students snap to the crucial connection between numeric encoding and our ability to use math to cull, filter and cluster data.  It’s a necessary precursor to their gaining Proustian “new eyes” for ESI.

Because ESI is just a bunch of numbers, we can use algorithms (mathematical formulas) to distill and compare those numbers.  Every student of electronic discovery learns about cryptographic hash functions and their usefulness as tools to digitally fingerprint files in support of identification, authentication, exclusion and deduplication.  When I teach law students about hashing, I tell them that hash functions are published, standard mathematical algorithms into which we input digital data of arbitrary size and the hash algorithm spits out a bit string (again, just a sequence of numbers) of fixed length called a “hash value.”  Hash values almost exclusively correspond to the digital data fed into the algorithm (termed “the message”) such that the chance of two different messages sharing the same hash value (called a “hash collision”) is exceptionally remote.  But because it’s possible, we can’t say each hash value is truly “unique.”

Using hash algorithms, any volume of data—from the tiniest file to the contents of entire hard drives and beyond—can be almost uniquely expressed as an alphanumeric sequence; in the case of the MD5 hash function, distilled to a value written as 32 hexadecimal characters (0-9 and A-F).  It’s hard to understand until you’ve figured out Base16; but, those 32 characters represent 340 trillion, trillion, trillion different possible values (2¹²⁸ or 16³²). Continue reading →