Preparing a talk about keyword search, I set out to distill observations gleaned from a host of misbegotten keyword search efforts, many from the vantage point of the court’s neutral expert née Special Master assigned to clean up the mess.  What emerged feels a bit…dark…and…uh…grouchy: like  truths no one wants to hear because then we might be obliged to change–when we all know how profitable it is to bicker about keywords in endless, costly rounds of meeting and conferring.  

The problems I’m dredging up have endured for decades, and their solutions have been clear and accessible for just as long.  So, why do we keep doing the same dumb things and expecting different outcomes?

In the 25+ years I’ve studied lexical search of ESI, I’ve learned that:

1. Lexical search is a crude tool that misses much more than it finds and leads to review of a huge volume of non-relevant information.  That said, even crude tools work wonders in the hands of skilled craftspeople who chip away with care to produce masterpieces.  The efficacy of lexical search increases markedly in the hands of adept practitioners who meticulously research, test and refine their search strategies.

2. Lawyers embrace lexical search despite knowing almost nothing about the limits and capabilities of search tools and without sufficient knowledge of the datasets and indices under scrutiny.  Grossly overestimating their ability to compose effective search queries, lawyers blithely proffer untested keywords and Boolean constructs.  Per Judge John Facciola a generation ago, lawyers think they’re experts in search “because they once used Google to find a Chinese restaurant in San Francisco that served dim sum and was open on Sundays.”

3. Without exception, every lexical search is informed and improved by the iterative testing of queries against a substantial dataset, even if that dataset is not the data under scrutiny.  Iterative testing is invaluable when queries are run against representative samples of the target data.  Every. Single. Time.

4. Hit counts alone are a poor measure of whether a lexical search is “good” or “bad.”  A “good” query may simply be generating an outsize hit count when run against the wrong dataset in the wrong way (e.g., searching for a person’s name in their own email).  Lawyers are too quick to exclude queries with high perceived hit counts before digging into the causes of poor precision.

5. A query’s success depends on how the dataset has been processed and indexed prior to search, challenging the assumption that search mechanisms just ‘work,’ as if by magic. 

6. Lexical search is a sloppy proxy for language; and language is replete with subtlety, ambiguity, polysemy and error, all serving to frustrate lexical search.  Effective lexical search adapts to accommodate subtlety, ambiguity, polysemy and error by, inter alia, incorporating synonyms, jargon and industry-specific language, common misspellings and alternate spellings (e.g., British vs. American spellings) and homophones, acronyms and initializations.

7. Lexical search’s utility lies equally in filtering out irrelevant data as it does in uncovering relevant information; so, it demands meticulous effort to mitigate the risk of overlooking pertinent documents.

Understanding some of these platitudes requires delving into the science of search and ESI processing.  A useful resource might be my 2019 primer on Processing in E-Discovery; admittedly not an easy read for all, but a window into the ways that processing ESI impacts searchability.

Fifteen years ago, I published a short paper called “Surefire Steps to Splendid Search” and set out ten steps that I promised would produce more effective, efficient and defensible queries.  Number 7 was:

“Test, Test, Test! The single most important step you can take to assess keywords is to test search terms against representative data from the universe of machines and data under scrutiny. No matter how well you think you know the data or have refined your searches, testing will open your eyes to the unforeseen and likely save a lot of wasted time and money.”

In the fullness of time, those ten steps ring as true today as when George Bush was in the White House. Then, as now, the greatest improvements in lexical search can be achieved with modest tweaks in methodology.  A stitch in time saves nine.

Another golden oldie is my 2012 collection of ten brief essays called “Shorties on Search.”

But, as much as I think those older missives hold up, and despite the likelihood that natural language prompts will soon displace old-school search queries, here’s a fresh recasting of my tips for better lexical search:

Essential Tips for Effective Lexical Search in Civil Discovery

Pre-Search Preparation:

1. Understand the Dataset
   • Identify data sources and types, then tailor the search to the data.
   • Assess the volume and organization of the dataset.  Can a search of fielded data facilitate improved precision?
   • Review any pre-processing steps applied, like normalization of case and diacriticals or use of stop words in creating the searchable indices.
2. Know Your Search Tools
   • Familiarize yourself with the tool’s syntax and keyword search capabilities.
   • Understand the tool’s limitations, especially with non-textual data and large documents.
3. Consult with Subject Matter Experts (SMEs)
   • Engage SMEs for insights on relevant terminology and concepts.
   • Use SME knowledge to refine keyword selection and search strategies.

Search Term Selection and Refinement:

4. Develop Comprehensive Keyword Lists
   • Include synonyms, acronyms, initializations, variants, and industry-specific jargon.
   • Consider linguistic and regional variations.
   • Account for misspellings, alternate spellings and common transposition errors.
5. Utilize Boolean Logic and Advanced Operators
   • Apply Boolean operators and proximity searches effectively.
   • Experiment with wildcards and stemming for broader term inclusion.
6. Iteratively Test and Refine Search Queries
   • Conduct sample searches to evaluate and refine search terms.
   • Adjust queries based on testing outcomes and new information.

Execution and Review:

7. Provide for Consistent Implementation Across Parties and Service Providers
   • Use agreed-upon terms where possible.  The most defensible search terms and methods are those the parties choose collaboratively.
   • Ensure consistency in search term application across the datasets, over time and among multiple parties.
8. Sample and Manually Review Results
   • Randomly sample search results to assess precision and recall.
   • Adjust search terms and strategies based on manual review findings.
9. Negotiate Search Terms with Opposing Counsel
   • Engage in discussions to agree on search terms and methodologies.
   • Document agreements to preempt disputes over discovery completeness.
   • Make abundantly clear whether a non-privileged document hit by a query must be produced or whether (as most producing parties assume) the items hit may nevertheless be withheld after a review for responsiveness. 

Post-Search Analysis:

10. Validate and Document the Search Process
    • Maintain comprehensive documentation of search terms, queries, exception items and decisions.  Never employ a set of queries to exclude items from discovery without the ability to document the queries and process employed.
    • Ensure the search methodology is defensible and compliant with legal standards.
11. Adapt and Evolve Search Strategies
    • Remain flexible to adapt strategies as case evidence and requirements evolve.
    • Leverage lessons from current searches to refine future discovery efforts.
12. Ensure Ethical and Legal Compliance
    • Adhere to privacy, privilege, and ethical standards throughout the discovery process.
    • Review and apply discovery protocols and court orders accurately.

For the second class meeting of my law school courses on E-Discovery and Digital Evidence, I require my students read the fourteen Sedona Conference Principles from the latest edition of “Best Practices, Recommendations & Principles for Addressing Electronic Document Production.” The Sedona principles are the bedrock of that group’s work on ESI and, notwithstanding my misgivings that the Principles have tilted toward blocking discovery more than guiding it, there’s much to commend in each of the three versions of the Principles released over the last twenty years.  They enjoy a constitutional durability in the eDiscovery community.

When my students read the Principles, I revisit them and each time, something jumps out at me.  This semester, it’s the musty language of Principle 9:

Principle 9: Absent a showing of special need and relevance, a responding party should not be required to preserve, review, or produce deleted, shadowed, fragmented, or residual electronically stored information.

The Sedona Principles, Third Edition: Best Practices, Recommendations & Principles for Addressing Electronic Document Production, 19 SEDONA CONF. J. (2018)

Save for the substitution of “electronically stored information” for the former “data or documents,” Principle 9 hasn’t been touched since its first drafts of 20+ years ago.  One could argue its longevity owes to an abiding wisdom and clarity. Indeed, the goals behind P9 are laudable and sound.  But the language troubles me, particularly the terms, “shadowed” and “fragmented,” which someone must have pulled out of their … I’ll say “hat” … during the Bush administration, and presumably no one said, “Wait, is that really a thing?”  In the ensuing decades, did no one question the wording or endeavor to fix it?

My objection is that both are terms of art used artlessly.  Consider “shadowed” ESI.  Run a search for shadowed ESI or data, and you’ll not hit anything on point but the Principle itself.  Examine the comments to Principle 9 and discover there’s no effort to explain or define shadowed ESI.  Head over to The Sedona Conference Glossary: eDiscovery and Digital Information Management, and you’ll find nary a mention of “shadowed” anything. 

That is not to say that there wasn’t a far-behind-the-scenes service existing in Microsoft Windows XP and Windows Server to facilitate access to locked files during backup that came to be called “Volume Shadow Copy Services” or “VSS,” but it wasn’t being used for forensics when the language of Principle 9 was floated.  I was a forensic examiner at the time and can assure you that my colleagues and I didn’t speak of “shadowed” data or documents.

But whether an argument can be made that it was a “thing” or not twenty years ago, it’s never been a term in common use, nor one broadly understood by lawyers and judges.  It’s not defined in the Principles or glossaries.  You’ll get no useful guidance from Google. 

What harm has it done?  None I can point to.  What good has it done?  None.  Yet, it might be time to consign “shadowed” to the dustbin of history and find something less vague.  It’s not gospel, it’s gobbledygook.

“Fragmented” is a term that’s long been used in reference to data storage, but not as a synonym for “residual” or “artifact.”  Fragmented files refer to information stored in non-contiguous clusters on a storage medium.  Many of the files we access and know to be readily accessible are fragmented in this fashion, and no one who understands the term in the context of ESI would confuse “fragmented” data or documents with something burdensome to retrieve.  But don’t take my word for that, Sedona’s own glossary backs me up.  Sedona’s Principle 9 doesn’t use “fragmented” as Sedona defines it.

If the drafters meant “fragments of data,” intending to convey “artifacts recoverable through computer forensics but not readily accessible to or comprehended by users,” then perhaps other words are needed, though I can’t imagine what those words would add that “deleted” or “residual” doesn’t cover.

This is small potatoes. No one need lose a wink of sleep over the sloppy wording, and I’m not the William Safire of e-discovery or digital forensics; but words matter.  When you are writing to guide persons without deep knowledge of the subject matter, your words matter very much.  If you use a term of art, make sure it’s a correct usage, a genuine one; and be certain you’ve either used it as experts do or define the anomalous usage in context.

When I fail to do that, Dear Reader, I hope you’ll call me on it, too.