Category Archives: Uncategorized

A Master Table of Truth

04 Tuesday Nov 2025

Posted by in ai, Computer Forensics, E-Discovery, General Technology Posts, Law Practice & Procedure, Uncategorized

≈ 5 Comments

Tags

, , , , , ,

Lawyers using AI keep turning up in the news for all the wrong reasons—usually because they filed a brief brimming with cases that don’t exist. The machines didn’t mean to lie. They just did what they’re built to do: write convincingly, not truthfully.

When you ask a large language model (LLM) for cases, it doesn’t search a trustworthy database. It invents one. The result looks fine until a human judge, an opponent or an intern with Westlaw access, checks. That’s when fantasy law meets federal fact.

We call these fictions “hallucinations,” which is a polite way of saying “making shit up;” and though lawyers are duty-bound to catch them before they reach the docket, some don’t. The combination of an approaching deadline and a confident-sounding computer is a dangerous mix.

Perhaps a Useful Guardrail

It struck me recently that the legal profession could borrow a page from the digital forensics world, where we maintain something called the NIST National Software Reference Library (NIST NSRL). The NSRL is a public database of hash values for known software files. When a forensic examiner analyzes a drive, the NSRL helps them skip over familiar system files—Windows dlls and friends—so they can focus on what’s unique or suspicious.

So here’s a thought: what if we had a master table of genuine case citations—a kind of NSRL for case citations?

Picture a big, continually updated, publicly accessible table listing every bona fide reported decision: the case name, reporter, volume, page, court, and year. When your LLM produces Smith v. Jones, 123 F.3d 456 (9th Cir. 2005), your drafting software checks that citation against the table.

If it’s there, fine—it’s probably references a genuine reported case.
If it’s not, flag it for immediate scrutiny.

Think of it as a checksum for truth. A simple way to catch the most common and indefensible kind of AI mischief before it becomes Exhibit A at a disciplinary hearing.

The Obstacles (and There Are Some)

Of course, every neat idea turns messy the moment you try to build it.

Coverage is the first challenge. There are millions of decisions, with new ones arriving daily. Some are published, some are “unpublished” but still precedential, and some live only in online databases. Even if we limited the scope to federal and state appellate courts, keeping the table comprehensive and current would be an unending job; but not an insurmountable obstacle.

Then there’s variation. Lawyers can’t agree on how to cite the same case twice. The same opinion might appear in multiple reporters, each with its own abbreviation. A master table would have to normalize all of that—an ambitious act of citation herding.

And parsing is no small matter. AI tools are notoriously careless about punctuation. A missing comma or swapped parenthesis can turn a real case into a false negative. Conversely, a hallucinated citation that happens to fit a valid pattern could fool the filter, which is why it’s not the sole filter.

Lastly, governance. Who would maintain the thing? Westlaw and Lexis maintain comprehensive citation data, but guard it like Fort Knox. Open projects such as the Caselaw Access Project and the Free Law Project’s CourtListener come close, but they’re not quite designed for this kind of validation task. To make it work, we’d need institutional commitment—perhaps from NIST, the Library of Congress, or a consortium of law libraries—to set standards and keep it alive.

Why Bother?

Because LLMs aren’t going away. Lawyers will keep using them, openly or in secret. The question isn’t whether we’ll use them—it’s how safely and responsibly we can do so.

A public master table of citations could serve as a quiet safeguard in every AI-assisted drafting environment. The AI could automatically check every citation against that canonical list. It wouldn’t guarantee correctness, but it would dramatically reduce the risk of citing fiction. Not coincidentally, it would have prevented most of the public excoriation of careless counsel we’ve seen.

Even a limited version—a federal table, or one covering each state’s highest court—would be progress. Universities, courts, and vendors could all contribute. Every small improvement to verifiability helps keep the profession credible in an era of AI slop, sloppiness and deep fakes.

No Magic Bullet, but a Sensible Shield

Let’s be clear: a master table won’t prevent all hallucinations. A model could still misstate what a case holds, or cite a genuine decision for the wrong proposition. But it would at least help keep the completely fabricated ones from slipping through unchecked.

In forensics, we accept imperfect tools because they narrow uncertainty. This could do the same for AI-drafted legal writing—a simple checksum for reality in a profession that can’t afford to lose touch with it.

If we can build databases to flag counterfeit currency and pirated software, surely we can build one to spot counterfeit law?

Until that day, let’s agree on one ironclad proposition: if you didn’t verify it, don’t file it.

Native or Not? Rethinking Public E-Mail Corpora for E-Discovery (Redux, 2013→2025)

16 Saturday Aug 2025

Posted by in ai, Computer Forensics, E-Discovery, Uncategorized

≈ 2 Comments

Tags

, , , , , , , ,

Yesterday, I found myself in a spirited exchange with a colleague about whether the e-discovery community has suitable replacements for the Enron e-mail corpora1—now more than two decades old—as a “sandbox” for testing tools and training students. I argued that the quality of the data matters: native or near-native e-mail collections remain essential to test processing and review workflows in ways that mirror real-world litigation.

The back-and-forth reminded me that, unlike forensic examiners or service providers, ediscovery lawyers may not know or care much about the nature of electronically-stored information until it finds its way to a review tool. I get that. If your interest in email is in testing AI coding tools, you’re laser-focused on text and maybe a handful of metadata; but if your focus is on the integrity and authenticity of evidence, or in perfecting processing tools, the originating native or near-native form of the corpus matters more.

What follows is a re-publication of a post from July 2013. I’m bringing it back because the debate over forms of email hasn’t gone away; the issue is as persistent and important as ever. A central takeaway bears repeating: the litmus test is whether a corpus hews to a fulsome RFC-5322 compliant format. If headers, MIME boundaries, and transport artifacts are stripped or incompletely synthesized, what remains ceases to be a faithful native or near-native format. That distinction matters, because even experienced e-discovery practitioners—those fixated on review at the far-right side of the EDRM—may not fully appreciate what an RFC-5322 email is, or how much fidelity is lost when working with post-processed sets.

Continue reading →

Adapting Requests for Production for AI GLLM Assessment

19 Monday Aug 2024

Posted by in Uncategorized

≈ 6 Comments

The integration of Generative Large Language Models (GLLMs) into the discovery process is transforming how documents are reviewed for relevance and responsiveness. These AI models, which excel at processing large document collections, offer significant efficiency improvements. However, to harness their full potential, requests for production (RFPs) must evolve to reflect the unique capabilities and limitations of AI systems. Traditional language in RFPs, which relies heavily on human intuition, needs to be adjusted to accommodate AI’s reliance on clear instructions, context, and precision. This post explores how to adapt requests for production to optimize GLLM usage, addressing both business disputes and tort claims. I’ll provide examples to illustrate how common RFPs can be refined for AI-assisted document review.

Effective AI Prompts in Discovery

AI systems like GLLMs function best with well-structured prompts. In the context of discovery, this means adjusting RFPs to emphasize clarity, specificity, and relevance. The key elements for constructing effective AI prompts in legal discovery are:

  1. Clarity and Specificity: Ambiguity can cause AI systems to miss important documents or misclassify irrelevant ones. Specific requests guide the AI more effectively.
  2. Contextual Guidance: AI relies on context to assess relevance. Providing additional background or specifying the purpose of certain requests helps refine the search.
  3. Keyword Precision: GLLMs rely on keywords to understand and evaluate document content. Choosing precise terms helps reduce the retrieval of irrelevant documents.
  4. Examples: AI systems can better identify relevant documents if examples are provided within the RFP, as they offer patterns for the system to follow.

Incorporating these principles into RFPs ensures that AI models can make the most accurate assessments during document review.

Adapting Requests for Production: Business Dispute and Tort Claim Examples

Both business disputes and tort claims involve unique types of documents and keywords. Adapting RFPs to suit GLLM’s capabilities involves providing detailed instructions for both.

Continue reading →

Girding for the E-Savvy Opponent (Revisited)

26 Friday Apr 2024

Posted by in Computer Forensics, E-Discovery, Uncategorized

≈ 7 Comments

Tags

, , , ,

A friend shared that she was seeing the Carole King musical, “Beautiful,” and I recalled the time I caught it twice on different visits to London in 2015 because I enjoyed it so. I reflected on why I was in London in Summer nine years ago and came across a post from the time–a post that I liked well-enough to revisit it below. I predicted the emergence of the e-savvy opponent, something that has indeed come to pass, yet less-widely or -effectively than I’d hoped (and still hope for). A new generation of e-discoverers has emerged since, so perhaps the post will be fresh (and remain relevant) for more than a few, and sufficiently forgotten to feel fresh for the rest:

(From May 12, 2015): I am in Great Britain this week addressing an E-Discovery and Information Governance conclave, joined by esteemed American colleagues and friends, Jason Baron and Ralph Losey among other luminaries.  My keynote topic opening the conference is Girding for the E-Savvy Opponent. Here is a smattering of what I expect to say.

I arrived in London from Budapest in time to catch some of the events for the 70th anniversary of VE Day, marking the hard-won victory over Germany in the war that shortly followed the war that was to have ended all wars.

As we sported poppies and stood solemnly at the Cenotaph recalling the sacrifices made by our parents and grandparents, I mulled technology’s role in battle, and the disasters that come from being unprepared for a tech-savvy opponent.

It’s said that, “Generals are always prepared to fight the last war.” This speaks as much to technology as to tactics.  Mounted cavalry proved no match for armored tanks.  Machine guns made trench warfare obsolete.  The Maginot Line became a punch line thanks to the Blitzkrieg. “Heavy fortifications?  “No problem, mein schatzi, ve vill just drive arount tem.”

In e-disclosure, we still fight the last war, smug in the belief that our opponents will never be e-savvy enough to defeat us.

Our old war ways have served so long that we are slow to recognize a growing vulnerability.  To date, our opponents have proved unsophisticated, uncreative and un-tenacious.  Oh, they make a feint against databases here and a half-hearted effort to get native production there; but, for the most part, they’re still fighting with hordes, horses and sabers.  We run roughshod over them.  We pacify them with offal and scraps.

But, we don’t think of it that way, of course.  We think we are great at all this stuff, and that the way we do things is the way it’s supposed to be done.  Large companies and big law firms have been getting away with abusive practices in e-disclosure for so long that they have come to view it as a birthright.  I am the 19th Earl of TIFF.  My father was the Royal Exchequer of Keywords.  I have more than once heard an opponent defend costly, cumbersome procedures that produce what I didn’t seek and didn’t want with the irrefutable justification of, “we did what we always do.”

Tech-challenged opponents make it easy.  They don’t appreciate how our arsenal of information has changed; so, they shoot at us with obsolete requests from the last war, the paper war.  They don’t grasp that the information they need now lives in databases and won’t be found by keywords.  They demand documents.  We have data.  They demand files.  We have sources.

Girding for the Tech Savvy Opponent-IQPC 2015

But, our once tech challenged opponents will someday evolve into Juris Doctor Electronicus.  When they do, here is some of what to expect from them:

E-savvy counsel succeeds not by overreaching but by insisting on mere competence—competent scope, competent processes and competent forms of production.  Good, not just good enough.

Your most effective defense against e-savvy counsel is the Luddite judge who applies the standards of his or her former law practice to modern evidence. Your best strategy here is to continue to expose young lawyers to outmoded practices so that when they someday take the bench they will also know no better way.

Another strategy against e-savvy counsel is to embed outmoded practices in the rules and to immunize incompetence against sanctions.

But these are stopgap strategies–mere delaying tactics.  In the final analysis, the e-savvy opponent needn’t fear retrograde efforts to limit electronic disclosure. Today, virtually all evidence is born electronically; consequently, senseless restrictions on electronic disclosure cannot endure unless we are content to live in a society where justice abides in purposeful ignorance of the evidence.  We have not fallen so, and we will not fall that far.

The e-savvy opponent’s most powerful ally is the jurist who can distinguish between the high cost and burden occasioned by poor information governance and the high cost and burden that flows from overreaching by incompetent requests.  Confronted with a reasonable request, this able judge will give you no quarter because your IG house is not in order.

E-savvy counsel well understands that claims like, “that’s gone,” “we can’t produce it that way” and “we searched thoroughly” rarely survive scrutiny.

It’s not that no enterprise can match the skills of the e-savvy opponent. It’s that so few have ever had to do so.  Counsel for producing parties haven’t had to be particularly e-savvy because opposing counsel rarely were.

Sure, you may have been involved in the Black Swan discovery effort–the catastrophic case where a regulator or judges compelled you to go far beyond your normal scope. But, is that sustainable? Could you do that on a regular basis if all of your opponents were e-savvy?

You may respond, “But we shouldn’t have to respond that way on a regular basis.” In fact, you should, because “e-savvy” in our opponents is something we must come to expect and because, if the opponent is truly e-savvy, their requests will likely smack of relevance and reasonableness.

Remember, the e-savvy opponent about which I warn is not the twit with a form or the wanker who’s simply trying to inflate the scope of the disclosure as a means to extort settlement.  They’re no match for you.  The e-savvy opponent to fear is the one who can persuade a court that the scope is appropriate and proportionate because it is, in fact, both.

Cloud Attachments: Versions and Purview

08 Monday Apr 2024

Posted by in Computer Forensics, E-Discovery, Uncategorized

≈ 6 Comments

Tags

, , , , ,

Last week, I dug into Cloud Attachments to email, probing the propensity of producing parties’ to shirk collection of linked documents.  Here, I want to discuss the versioning concern offered as a justification for non-production and the use of hash duplicate identification to integrate supplementary productions with incomplete prior productions. 

Recently on LinkedIn, Very Smart Guy, Rachi Messing, shared this re: cloud attachments,

the biggest issue at hand is not the technical question of how to collect them and search them, but rather what VERSION is the correct one to collect and search.

Is it:

1. The version that existed at the time the email was sent (similar to a point in time capture of a file that is attached to an email the traditional way)

2. The version that was seen the first time the recipient opened it (which may lead to multiple versions required based on the exact timing of multiple recipients opening at varying times)

3. The version that exists the final time a recipient opened it

4. The most recent version in existence

I understand why Rachi might minimize the collection and search issue. He’s knee deep in Microsoft M365 collection.  As I noted in my last post, Microsoft makes cloud attachment collection a feature available to its subscribers, so there’s really no excuse for the failure to collect and search cloud attachments in Microsoft M365. 

I’d reframe Rachi’s question: Once collected, searched and determined to be responsive, is the possibility that the version of a cloud attachment reviewed differs from the one transmitted a sufficient basis upon which to withhold the attachment from production?

Respecting the versioning concern, I responded to Rachi’s post this way:

The industry would profit from objective analysis of the instance (e.g., percentage) of Cloud attachments modified after transmittal. I expect it will vary from sector to sector, but we would benefit from solid metrics in lieu of the anecdotal accounts that abound. My suspicion is that the instance is modest overall, the majority of Cloud attachments remaining static rather than manifesting as collaborative documents. But my suspicion would readily yield to meaningful measurement.  … May I add that the proper response to which version to collect to assess relevance is not ‘none of them,’ which is how many approach the task.

Digging into the versioning issue demands I retread ground on cloud attachments generally.

A “Cloud Attachment” is what Microsoft calls a file transmitted via email in which the sender places the file in a private online repository (e.g., Microsoft OneDrive) and sends a link to the uploaded file to the intended recipients.  The more familiar alternative to linking a file as a cloud attachment is embedding the file in the email; accordingly, such “Embedded Attachments” are collected with the email messages for discovery and cloud attachments are collected (downloaded) from OneDrive, ideally when the email is collected for discovery.  As a rule-of-thumb, large files tend to be cloud attachments automatically uploaded by virtue of their size.  The practice of linking large files as cloud attachments has been commonplace for more than a decade.

Within the Microsoft M365 email environment, searching and collecting email, including its embedded and cloud attachments, is facilitated by a suite of features called Microsoft Purview.  Terming any task in eDiscovery “one-click easy” risks oversimplification, but the Purview eDiscovery (Premium “E5”) features are designed to make collection of cloud attachments to M365 email nearly as simple as ticking a box during collection.

When a party using Microsoft M365 email elects to collect (export) a custodian’s email for search, they must decide whether to collect files sent as cloud attachments so they may be searched as part of the message “family,” the term commonly applied to a transmitting message and its attachments.  Preserving this family relationship is important because the message tells you who received the attachments and when, where searching the attachments tells you what information was shared. The following screenshot from Microsoft illustrates the box checked to collect cloud attachments. Looks “one-click easy,” right?

By themselves, the cloud attachment links in a message reveal nothing about the content of the cloud attachments.  Sensibly, the target documents must be collected to be assessed and as noted, the reason they are linked is not because they have some different character in terms of their relevance; many times they are linked because they are larger files, so to that extent, they hold a greater volume of potentially relevant information.

Just as it would not have been reasonable in the days of paper discovery to confine a search to documents on your desk but not in your desk, it’s not reasonable to confine a search of email attachments to embedded attachments but not cloud attachments.  Both are readily accessible to the custodians of the email using the purpose-built tools Microsoft supplies to its email customers.

Microsoft Purview collects cloud attachments as they exist at the time of collection; so, if the attachment was edited after transmittal, the attachment will reflect those edits.  The possibility that a document has been edited is not a new one in discovery; it goes to the document’s admissibility not its discoverability.  The relevance of a document for discovery depends on its content and logical unitization, and assessing content demands that it be searched, not ignored on the speculative possibility that it might have changed.

If a cloud attachment were changed after transmittal, those changes are customarily tracked within the document.  Accordingly, if a cloud attachment has run the gauntlet of search and review, any lingering suspicion that the document was changed may be resolved by, e.g., production of the version closest in time to transmittal or by the parties meeting and conferring.  Again, the possibility that a document has been edited is nothing new; and is merely a possibility.  It’s ridiculous to posit that a party may eschew collecting or producing all cloud attachments because some might have been edited.

Cloud attachments are squarely within the ambit of what must be assessed for relevance. The potential for a cloud attachment to be responsive is no less than that of an item transmitted as an embedded attachment.  The burden claimed by responding parties grows out of their failure to do what clearly should have been done in the first place; that is, it stems from the responding party’s decision to exclude potentially relevant, accessible documents from being collected and searched. 

If you’re smart, Dear Reader, you won’t fail to address cloud attachments explicitly in your proposed ESI Protocols and/or Requests for Production.  I can’t make this point too strongly, because you’re not likely to discover that the other side didn’t collect and search cloud attachments until AFTER they make a production, putting you in the unenviable posture of asking for families produced without cloud attachments to be reproduced with cloud attachments.  Anytime a Court hears that you are asking for something to be produced a second time in discovery, there’s a risk the Court may be misled by an objection grounded on Federal Rule of Civil Procedure Rule 34(b)(2)(E)(iii), which states that, [a] party need not produce the same electronically stored information in more than one form.”  In my mind, “incomplete” and “complete” aren’t what the drafters of the Rule meant by “more than one form,” but be prepared to rebut the claim.

At all events, a party who failed to collect cloud attachments will bewail the need to do it right and may cite as burdensome the challenge of distinguishing items reviewed without cloud transmittals from those requiring review when made whole by the inclusion of cloud attachments.

Once a party collects cloud attachments and transmittals, there are various ways to distinguish between messages updated with cloud attachments and those previously reviewed without cloud attachments.  Identifying families previously collected that have grown in size is one approach.  Then, by applying a filter, only the attachments of these families would be subjected to supplementary keyword search and review.  The emails with cloud attachments that are determined to be responsive and non-privileged would be re-produced as families comprising the transmittal and all attachments (cloud AND embedded).  An overlay file may be used to replace items previously produced as incomplete families with complete families.  No doubt there are other efficient approaches.

If all transmittal messages were searched and assessed previously (albeit without their cloud attachments), there would not be a need to re-assess those transmittals unless they have become responsive by virtue of a responsive cloud attachment.  These “new” families need no de-duplication against prior production because they were not produced previously.  I know that sounds peculiar, but I promise it makes sense once you think through the various permutations.

With respect to using hash deduplication, the hash value of a transmittal does not change because you collect a NON-embedded cloud attachment; leastwise not unless you change the way you compute the hash value to incorporate the collected cloud attachment.  Hash deduplication of email has always entailed the hashing of selected components of messages because email headers vary.  Accordingly, a producing party need compare only the family segments that matter, not the ones that do not. In other words, de-duplicating what has been produced versus new material is a straightforward process for emails (and one that greatly benefits from use of the EDRM MIH). Producing parties do not need to undertake a wholesale re-review of messages; instead, they need to review for the first time those things they should have reviewed from inception.

I’ll close with a question for those who conflate cloud attachments (which reside in private cloud respositories) with hyperlinks to public-facing web resources, objecting that dealing with collecting cloud attachments will require collection of all hyperlinked content. What have you been doing with the hyperlinks in your messages until now? In my experience, loads of us include a variety of hyperlinks in email signature blocks. We’ve done it for years. In my email signature, I hyperlink to my email address, my website and my blog; yet, I’ve never had trouble distinguishing those links from embedded and cloud attachments. The need to integrate cloud attachments in eDiscovery is not a need to chase every hyperlink in an email. Doug Austin does a superb job debunking the “what about hyperlinks” strawman in Assumption One of his thoughtful post, “Five Assumptions About the Issue of Hyperlinked Files as Modern Attachments.”

Bottom Line: If you’re an M365 email user; you need to grab the cloud attachments in your Microsoft repositories. If you’re a GMail user, you need to grab the cloud attachments in your Google Drive respositories. That a custodian might conceivably link to another repository is no reason to fail to collect from M365 and GMail.

Image

What’s All the Fuss About Linked Attachments?

29 Friday Mar 2024

Tags

, , ,

In the E-Discovery Bubble, we’re embroiled in a debate over “Linked Attachments.” Or should we say “Cloud Attachments,” or “Modern Attachments” or “Hyperlinked Files?” The name game aside, a linked or Cloud attachment is a file that, instead of being tucked into an email, gets uploaded to the cloud, leaving a trail in the form of a link shared in the transmitting message. It’s the digital equivalent of saying, “It’s in an Amazon locker; here’s the code” versus handing over a package directly.  An “embedded attachment” travels within the email, while a “linked attachment” sits in the cloud, awaiting retrieval using the link.

Some recoil at calling these digital parcels “attachments” at all. I stick with the term because it captures the essence of the sender’s intent to pass along a file, accessible only to those with the key to retrieve it, versus merely linking to a public webpage.  A file I seek to put in the hands of another via email is an “attachment,” even if it’s not an “embedment.” Oh, and Microsoft calls them “Cloud Attachments,” which is good enough for me.

Regardless of what we call them, they’re pivotal in discovery. If you’re on the requesting side, prepare for a revelation. And if you’re a producing party, the party’s over.

A Quick March Through History

Nascent email conveyed basic ASCII text but no attachments.  In the early 90s, the advent of Multipurpose Internet Mail Extensions (MIME) enabled files to hitch a ride on emails via ASCII encoded in Base64. This tech pivot meant attachments could join emails as encoded stowaways, to be unveiled upon receipt.

For two decades, this embedding magic meant capturing an email also netted its attachments. But come the early 2010s, the cloud era beckoned. Files too bulky for email began diverting to cloud storage with emails containing only links or “pointers” to these linked attachments. 

The Crux of the Matter

Linked attachments aren’t newcomers; they’ve been lurking for over a decade. Yet, there’s a growing “aha” moment among requesters as they realize the promised exchange of digital parcels hasn’t been as expected. Increasingly—and despite contrary representations by producing parties—relevant, responsive and non-privileged attachments to email aren’t being produced because relevant, responsive and non-privileged attachments aren’t being searched.

Wait! What?  Say that again.

You heard me.  As attachments shifted from being embedded to being linked, producing parties simply stopped collecting and searching those attachments.

How is that possible?  Why didn’t they disclose that? 

I’ll explain if you’ll indulge me in another history lesson.

Echoes From the Past

Traditionally, discovery leaned on indexing the content of email and attachments for quicker search, bypassing the need to sift through each individually.  Every service provider employs indexed search. 

When attachments are embedded in messages, those attachments are collected with the messages, then indexed and searched.  But when those attachments are linked instead of embedded, collecting them requires an added step of downloading the linked attachments with the transmitting message.  You must do this before you index and search because, if you fail to do so, the linked attachments aren’t searched or tied to the transmitting message in a so-called “family relationship.”

They aren’t searched.  Not because they are immaterial or irrelevant or in any absolute sense, inaccessible; a linked attachment is as amenable to being indexed and searched as any other document. They aren’t searched because they aren’t collected; and they aren’t collected because it’s easier to blow off linked attachments than collect them.

Linked attachments, squarely under the producer’s control, pose a quandary. A link in an email is a dead-end for anyone but the sender and recipients and reveals nothing of the file’s content. These linked attachments could be brimming with relevant keywords yet remain unexplored if not collected with their emails.

So, over the course of the last decade, how many times has an opponent revealed that, despite a commitment to search a custodian’s email, they were not going to collect and search linked documents?

The curse and blessing of long experience is having seen it all before.  Every generation imagines they invented sex, drugs and rock-n-roll, and every new information and communication technology is followed by what I call the “getting-away-with-murder” phase in civil discovery.  Litigants claim that whatever new tech has wrought is “too hard” to deal with in discovery, and they get away with murder by not having to produce the new stuff until long after we have the means and methods to do so.  I lived through that with e-mail, native production, then mobile devices, web content and now, linked attachments.

This isn’t just about technology but transparency and diligence in discovery. The reluctance to tackle linked attachments under claims of undue burden echoes past reluctances with emerging technologies. Yet, linked attachments, integral to relevance assessments, shouldn’t be sidelined.

What is the Burden, Really?

We see conclusory assertions of burden notwithstanding that the biggest platforms like Microsoft and Google offer ‘pretty good’ mechanisms to deal with linked attachments.  So, if a producing party claims burden, it behooves the Court and requesting parties to inquire into the source of the messaging.  When they do, judges may learn that the tools and techniques to collect linked attachments and preserve family relationships exist, but the producing party elected not to employ them.  Granted, these tools aren’t perfect; but they exist, and perfect is not the standard, just as pretending there are no solutions and doing nothing is not the standard. 

Claims that collecting linked attachments pose an undue burden because of increased volume are mostly nonsense.  The longstanding practice has been to collect a custodian’s messages and ALL embedded attachments, then index and search them.  With few exceptions, the number of items collected won’t differ materially whether the attachment is embedded or linked (although larger files tend to be linked).  So, any party arguing that collecting linked attachments will require the search of many more documents than before is fibbing or out of touch.  I try not to attribute to guile that which may be explained by ignorance, so let’s go with the latter.

Half Baked Solutions

Challenged for failing to search linked attachments, a responding party may protest that they searched the transmitting emails and even commit to collecting and searching linked attachments to emails containing search hits.  Sounds reasonable, right?  Yet, it’s not even close to reasonable. Here’s why:

When using lexical (e.g., keyword) search to identify potentially responsive e-mail “families,” the customary practice is to treat a message and its attachments as potentially responsive if either the content of the transmitting message or its attachment generates search “hits” for the keywords and queries run against them.  This is sensible because transmittals often say no more than, “see attached;” it’s the attachment that holds the hits.  Yet, stripped of its transmittal, you won’t know the timing or circulation of the attachment. So, we preserve and disclose email families.

But, if we rely upon the content of transmitting messages to prompt a search of linked attachments, we will miss the lion’s share of responsive evidence.  If we produce responsive documents without tying them to their transmittals, we can’t tell who got what and when.  All that “what did you know and when did you know it” matters.

Why Guess When You Can Measure?

Hopefully, you’re wondering how many hits suggesting relevance occur in transmittals and how many in attachments?  How many occur in both?  Great questions!  Happily, we can measure these things.  We can determine, on average, the percentage of messages that produce hits versus their attachments. 

If you determine that, say, half of hits were within embedded attachments, then you can fairly attribute that character to linked attachments not being searched.  In that sense, you can estimate how much you’re missing and ascertain a key component of a proper proportionality analysis.

So why don’t producing parties asserting burden supply this crucial metric? 

The Path Forward

Producing parties have been getting away with murder on linked attachments for so long that they’ve come to view it as an entitlement. Linked attachments are squarely within the ambit of what must be assessed for relevance.  The potential for a linked attachment to be responsive is no less than that of an item transmitted as an embedded attachment.  So, let’s stop pretending they have a different character in terms of relevance and devote our energies to fixing the process.

Collecting linked attachments isn’t as Herculean as some claim, especially with tools from giants like Microsoft and Google easing the process. The challenge, then, isn’t in the tools but in the willingness to employ them.

Do linked attachments pose problems?  They absolutely do!  I’ve elided over ancillary issues of versioning and credentials because those concerns reside in the realm between good and perfect solutions. Collection methods must be adapted to them—with clumsy workarounds at first and seamless solutions soon enough.  But in acknowledging that there are challenges, we must also acknowledge that these linked attachments have been around for years, and they are evidence.  Waiting until the crisis stage to begin thinking about how to deal with them was a choice, and a poor one.  I shudder to think of the responsive information ignored every single day because this issue is inadequately appreciated by counsel and courts.

Happily, this is simply a technical challenge and one starting to resolve.  Speeding the race to resolution requires that courts stop giving a free pass to the practice of ignoring linked attachments.  Abraham Lincoln defined a hypocrite as a “man who murdered his parents, and then pleaded for mercy on the grounds that he was an orphan.”  Having created the problem and ignored it for years, it seems disingenuous to indulge requesting parties’ pleas for mercy.  

In Conclusion

We’re at a crossroads, with technical solutions within reach and the legal imperative clearer than ever. It’s high time we bridge the gap between digital advancements and discovery obligations, ensuring that no piece of evidence, linked or embedded, escapes scrutiny.

Posted by | Filed under Computer Forensics, E-Discovery, Uncategorized

≈ 18 Comments

ESI Protocols: How Do I Get Out of a Bad Deal?

19 Tuesday Mar 2024

Posted by in Uncategorized

≈ 4 Comments

Tags

,

I watched a webinar this morning where the presenters addressed ESI Protocols.  They were well-informed people sharing sound advice; but it underscored for me why people despise lawyers.  A presenter counseled, “Always build an escape clause into whatever you agree to.”

The speaker meant, if you commit your clients to a protocol provision, and you later find that the client or its service provider can’t or won’t do what was promised, you need to incorporate a “fingers crossed” way to back out of the deal.

Many readers—lawyer readers certainly—will count that as inspired advice. They’ll posit, “Aren’t we protecting our clients when we spare them the hardships of an improvident agreement?”  In truth, the risk of being bound by obligations that could prove more onerous or expensive than anticipated is the number one objection I hear voiced when I advocate for use of ESI Protocols.

Who wouldn’t want to walk out on their obligations when the going gets rough?  It’s human nature to crave the benefits of a bargain without its burdens; but just try to run a restaurant where everyone walks the check!

The law has a term for what accounts for the difference between a fair deal and a debacle: it’s due diligence.  Competent counsel should know the capabilities of both clients and vendors before agreeing to an obligation that hinges on the capabilities of our clients and vendors. 

Counsel who agrees to something because he didn’t understand the implications of the agreement won’t want to own that.  He will point the finger at anyone and everyone except himself.  That, too, is human nature, albeit not a pretty predilection.  But let’s face facts: Those lawyers weren’t tricked; they were uninformed and unprepared.

That said, not all unforeseen consequences of an ESI Protocol grow out a lack of diligence or competence.

People make mistakes. You do.  I do.  And when we do, the question becomes: Who should bear the brunt of our mistakes?  And when should the consequences of our mistakes be limited by proportionality and (for lack of a better term) mercy?

Long before I became an attorney, some canny counsel decided that the optimum legal advice to a culpable client was to admit nothing, don’t apologize, deny, deny, deny and mount a strong offense as your best defense. Perhaps that’s why lawyers are the last bastion of characters cast as vile stereotypes in the movies without outcry.  Okay, lawyers and Nazis.

If experience means anything, mine suggests that what passes for good legal advice is lousy life advice.  If you made an honest mistake in agreeing to a provision of an ESI protocol, the optimum path is to own it and seek to make it right.  Sometimes your opponent will relate and work decently to renegotiate the terms. Often, the Court will come to your aid if it’s clear you made a good faith mistake and you own it. Rarely, exceptionally, your client must endure some hardship for the error. 

In every case I’ve come across in the last 42 years, that final, onerous outcome coincided with a profound lack of competence or diligence when the deal was struck, the poster child being In re Fannie Mae Sec. Litig., 552 F.3d 814 (D.C. Cir. 2009), but also a line of cases where it’s hard to explain the outcome save for the absence of due diligence, e.g., McCormick & Co., Inc. v. Ryder Integrated Logistics, Inc., ___ F.Supp.3d ___, 2023 WL 2433902 (D. Md. March 9, 2023)

We speak reverently about “the Rule of Law;” but that rule begins within each of us, in our character and commitment.  The notion of including a clause in agreements to escape obligations when they become inconvenient is troubling and erodes the integrity of agreements and the foundations of a functioning society.

Lessons from Lousy Lexical Search (and Tips to Do Better)

26 Monday Feb 2024

Posted by in Computer Forensics, E-Discovery, Uncategorized

≈ 7 Comments

Preparing a talk about keyword search, I set out to distill observations gleaned from a host of misbegotten keyword search efforts, many from the vantage point of the court’s neutral expert née Special Master assigned to clean up the mess.  What emerged feels a bit…dark…and…uh…grouchy: like  truths no one wants to hear because then we might be obliged to change–when we all know how profitable it is to bicker about keywords in endless, costly rounds of meeting and conferring.  

The problems I’m dredging up have endured for decades, and their solutions have been clear and accessible for just as long.  So, why do we keep doing the same dumb things and expecting different outcomes?

In the 25+ years I’ve studied lexical search of ESI, I’ve learned that:

1. Lexical search is a crude tool that misses much more than it finds and leads to review of a huge volume of non-relevant information.  That said, even crude tools work wonders in the hands of skilled craftspeople who chip away with care to produce masterpieces.  The efficacy of lexical search increases markedly in the hands of adept practitioners who meticulously research, test and refine their search strategies.

2. Lawyers embrace lexical search despite knowing almost nothing about the limits and capabilities of search tools and without sufficient knowledge of the datasets and indices under scrutiny.  Grossly overestimating their ability to compose effective search queries, lawyers blithely proffer untested keywords and Boolean constructs.  Per Judge John Facciola a generation ago, lawyers think they’re experts in search “because they once used Google to find a Chinese restaurant in San Francisco that served dim sum and was open on Sundays.”

3. Without exception, every lexical search is informed and improved by the iterative testing of queries against a substantial dataset, even if that dataset is not the data under scrutiny.  Iterative testing is invaluable when queries are run against representative samples of the target data.  Every. Single. Time.

4. Hit counts alone are a poor measure of whether a lexical search is “good” or “bad.”  A “good” query may simply be generating an outsize hit count when run against the wrong dataset in the wrong way (e.g., searching for a person’s name in their own email).  Lawyers are too quick to exclude queries with high perceived hit counts before digging into the causes of poor precision.

5. A query’s success depends on how the dataset has been processed and indexed prior to search, challenging the assumption that search mechanisms just ‘work,’ as if by magic. 

6. Lexical search is a sloppy proxy for language; and language is replete with subtlety, ambiguity, polysemy and error, all serving to frustrate lexical search.  Effective lexical search adapts to accommodate subtlety, ambiguity, polysemy and error by, inter alia, incorporating synonyms, jargon and industry-specific language, common misspellings and alternate spellings (e.g., British vs. American spellings) and homophones, acronyms and initializations.

7. Lexical search’s utility lies equally in filtering out irrelevant data as it does in uncovering relevant information; so, it demands meticulous effort to mitigate the risk of overlooking pertinent documents.

Understanding some of these platitudes requires delving into the science of search and ESI processing.  A useful resource might be my 2019 primer on Processing in E-Discovery; admittedly not an easy read for all, but a window into the ways that processing ESI impacts searchability.

Fifteen years ago, I published a short paper called “Surefire Steps to Splendid Search” and set out ten steps that I promised would produce more effective, efficient and defensible queries.  Number 7 was:

“Test, Test, Test! The single most important step you can take to assess keywords is to test search terms against representative data from the universe of machines and data under scrutiny. No matter how well you think you know the data or have refined your searches, testing will open your eyes to the unforeseen and likely save a lot of wasted time and money.”

In the fullness of time, those ten steps ring as true today as when George Bush was in the White House. Then, as now, the greatest improvements in lexical search can be achieved with modest tweaks in methodology.  A stitch in time saves nine.

Another golden oldie is my 2012 collection of ten brief essays called “Shorties on Search.”

But, as much as I think those older missives hold up, and despite the likelihood that natural language prompts will soon displace old-school search queries, here’s a fresh recasting of my tips for better lexical search:

Essential Tips for Effective Lexical Search in Civil Discovery

Pre-Search Preparation:

  1. Understand the Dataset
    • Identify data sources and types, then tailor the search to the data.
    • Assess the volume and organization of the dataset.  Can a search of fielded data facilitate improved precision?
    • Review any pre-processing steps applied, like normalization of case and diacriticals or use of stop words in creating the searchable indices.
  2. Know Your Search Tools
    • Familiarize yourself with the tool’s syntax and keyword search capabilities.
    • Understand the tool’s limitations, especially with non-textual data and large documents.
  3. Consult with Subject Matter Experts (SMEs)
    • Engage SMEs for insights on relevant terminology and concepts.
    • Use SME knowledge to refine keyword selection and search strategies.

Search Term Selection and Refinement:

  1. Develop Comprehensive Keyword Lists
    • Include synonyms, acronyms, initializations, variants, and industry-specific jargon.
    • Consider linguistic and regional variations.
    • Account for misspellings, alternate spellings and common transposition errors.
  2. Utilize Boolean Logic and Advanced Operators
    • Apply Boolean operators and proximity searches effectively.
    • Experiment with wildcards and stemming for broader term inclusion.
  3. Iteratively Test and Refine Search Queries
    • Conduct sample searches to evaluate and refine search terms.
    • Adjust queries based on testing outcomes and new information.

Execution and Review:

  1. Provide for Consistent Implementation Across Parties and Service Providers
    • Use agreed-upon terms where possible.  The most defensible search terms and methods are those the parties choose collaboratively.
    • Ensure consistency in search term application across the datasets, over time and among multiple parties.
  2. Sample and Manually Review Results
    • Randomly sample search results to assess precision and recall.
    • Adjust search terms and strategies based on manual review findings.
  3. Negotiate Search Terms with Opposing Counsel
    • Engage in discussions to agree on search terms and methodologies.
    • Document agreements to preempt disputes over discovery completeness.
    • Make abundantly clear whether a non-privileged document hit by a query must be produced or whether (as most producing parties assume) the items hit may nevertheless be withheld after a review for responsiveness. 

Post-Search Analysis:

  1. Validate and Document the Search Process
    • Maintain comprehensive documentation of search terms, queries, exception items and decisions.  Never employ a set of queries to exclude items from discovery without the ability to document the queries and process employed.
    • Ensure the search methodology is defensible and compliant with legal standards.
  2. Adapt and Evolve Search Strategies
    • Remain flexible to adapt strategies as case evidence and requirements evolve.
    • Leverage lessons from current searches to refine future discovery efforts.
  3. Ensure Ethical and Legal Compliance
    • Adhere to privacy, privilege, and ethical standards throughout the discovery process.
    • Review and apply discovery protocols and court orders accurately.

Will AI Summarization Disrupt Discovery?

26 Friday Jan 2024

Posted by in Uncategorized

≈ 6 Comments

Tags

, ,

Reader’s Digest, the century-old magazine with the highest paid circulation, has long published “condensed” books; anthologies of four-to-five popular novels abridged to fit in a single volume.  Condensed Books were once enormously popular, with tens of millions of copies in circulation.  They were also an abomination to serious readers, a literary Tang for those who preferred fresh-squeezed OJ. I’ve never read a condensed book, so I’m in no position to judge their merit save to say that I believe reading anything is a good thing.  I imagine the condensed versions conveyed the guts of the story well enough to sound like you’d read it over drinks with the neighbors before the Ed Sullivan show.

But I am enough of a purist (okay, “snob”) to worry about the impact of summarization.  As an undergraduate English major, I had to wade through some challenging tomes.  I have no empirical evidence for it, but I’m certain those books are a part of me in ways they never would have been had I sought out the Cliffs Notes instead.  I expect most avid readers feel the same.  Summaries necessarily discard content, and what remains is incapable of conveying the same tone, nuance and detail.

So, I worry when the tech industry touts the value of AI summarization of documents, especially as a means of speeding identification and review of evidence in discovery.  I question whether the “Reader’s Digest Condensed Evidence” will convey the same tone, nuance and detail that characterize responsive productions.  Will distillation be made of distillations until genuine intelligence is lost altogether? 

It’s an inchoate apprehension—an old man’s anxiety perhaps—but litigation is about human behavior, human frailty and failings.  I fear too much humanity will disappear in AI-generated summaries with the underlying communications less likely to see the light of day.  The mandate that discovery be “just, speedy and inexpensive” is now read as “just speedy and inexpensive.”  That discarded comma is tragic.

Technology is my lifelong passion.  So, I am not afraid of new tech as much as put off by the embrace of technology to further speed and economy without due consideration of quality.  LegalWeek 2024 will be a carnival of vendors touting AI features and roadmaps.  How many will have metrics to support the quality of their AI-abetted outcomes?  How many have forgotten the comma while chasing the cash? Per Upton Sinclair, ““It is difficult to get a man to understand something, when his salary depends on his not understanding it.”

Unquestionably, we must reduce the cost of discovery to protect the portals of justice.  Justice no one can afford to pursue is no justice at all.  But there are uniquely human characteristics we should continue to esteem in discovery, like curiosity, intuition, suspicion and impression; the “Spidey-sense” we derive from tone, nuance and detail.  Before we use AI to summarize collections then deploy AI to characterize the summaries, can we pause just long enough to see if it’s going to work? Real testing, not just that which supports salaries.

Policy for Student Use of AI

11 Monday Dec 2023

Posted by in Uncategorized

≈ 8 Comments

Tags

, , , , , ,

Andy Williams used to croon that this is “The Most Wonderful Time of the Year.” For me, it’s time to update the curriculum for my class on Electronic Discovery and Digital Evidence at the University of Texas in the graduate schools of Law, Computer Science and Information Science. I’ve long built the course around a Workbook I wrote with readings and some two dozen exercises. But, when I last taught the course a year ago, generative AI was hardly a twinkle in Santa’s eye. Now, of course, AI is the topic that’s eaten all others. So, I’ve had to fashion a policy for student use of AI. I elected to embrace student use of AI tools, in part because legal scholarsip is artful plagarism termed “precedent” and–let’s face it–students are going to use LLMs, whatever I say. So, here’s what I’ve come up with. I’ll be grateful for your feedback as comments, most especially if you are an educator facing the same issues with advice born of experience.

Use of Generative Large Language Models to Assist with Exercises

1. Explicit Disclosure Requirement

  • It is a violation of the honor code to misrepresent work by characterizing it as your own if it is not.  Students may use generative LLMs, such as ChatGPT or Bard, for assistance in completing Workbook exercises; however, they must explicitly disclose the use of these tools by providing a brief note or acknowledgment in their submissions. Transparency is mandatory.

2. Verification and Cross-Checking

  • Students may utilize generative LLMs during Workbook exercises but are required to independently verify and cross-check the information generated by these models through additional research using alternate, reliable sources.

3.  Accountability

  • While generative LLMs are permitted tools, students are held accountable for the accuracy and completeness of the information obtained from these models. Any errors or omissions resulting from the use of LLMs are considered the responsibility of the student. This policy underscores the importance of independent verification and personal accountability.

4. Prohibited for Quizzes and Exams

  • Notwithstanding the foregoing, you may not consult any source of information, including AI resources, when completing quizzes or the final exam.

POSTSCRIPT: I add this a day after the foregoing, after reading that the Fifth Circuit’s proposed a rule change requiring that counsel and pro se litigants certify of any filed document, that “no generative artificial intelligence program was used in drafting the document…or to the extent such a program was used, all generated text, including all citations and legal analysis, has been reviewed for accuracy and approved by a human.” I recall shaking my head at how foolish it was when a grandstanding district court judge made headlines by requiring such certifications following a high-profile gaffe in New York. “Of course a lawyer must verify the accuracy of legal analysis and citations! Lawyers shouldn’t need to certify that we did what we are required to do!”

Yet, here I am requiring my students to do much the same. I feel confident in advising students that, if they use AI, they must verify the information and sink or swim based on what they submit, even if the AI hallucinates or misleads. Back in the day, lawyers knew they had to “Shepardize” citations to verify that the cases cited were still solid. Proffering a a made-up citation was beyond comprehension.

So, am I right to require explicit disclosure of generative AI? Or will AI soon be woven into so many sources of information that disclosure will feel as foolish as requiring students to disclose they used a word processor instead of a typewriter would have been forty years ago? I’m struggling with this. What do you think?