Tag Archives: generative-ai

Will AI Summarization Disrupt Discovery?

26 Friday Jan 2024

Posted by in Uncategorized

≈ 6 Comments

Tags

, ,

Reader’s Digest, the century-old magazine with the highest paid circulation, has long published “condensed” books; anthologies of four-to-five popular novels abridged to fit in a single volume.  Condensed Books were once enormously popular, with tens of millions of copies in circulation.  They were also an abomination to serious readers, a literary Tang for those who preferred fresh-squeezed OJ. I’ve never read a condensed book, so I’m in no position to judge their merit save to say that I believe reading anything is a good thing.  I imagine the condensed versions conveyed the guts of the story well enough to sound like you’d read it over drinks with the neighbors before the Ed Sullivan show.

But I am enough of a purist (okay, “snob”) to worry about the impact of summarization.  As an undergraduate English major, I had to wade through some challenging tomes.  I have no empirical evidence for it, but I’m certain those books are a part of me in ways they never would have been had I sought out the Cliffs Notes instead.  I expect most avid readers feel the same.  Summaries necessarily discard content, and what remains is incapable of conveying the same tone, nuance and detail.

So, I worry when the tech industry touts the value of AI summarization of documents, especially as a means of speeding identification and review of evidence in discovery.  I question whether the “Reader’s Digest Condensed Evidence” will convey the same tone, nuance and detail that characterize responsive productions.  Will distillation be made of distillations until genuine intelligence is lost altogether? 

It’s an inchoate apprehension—an old man’s anxiety perhaps—but litigation is about human behavior, human frailty and failings.  I fear too much humanity will disappear in AI-generated summaries with the underlying communications less likely to see the light of day.  The mandate that discovery be “just, speedy and inexpensive” is now read as “just speedy and inexpensive.”  That discarded comma is tragic.

Technology is my lifelong passion.  So, I am not afraid of new tech as much as put off by the embrace of technology to further speed and economy without due consideration of quality.  LegalWeek 2024 will be a carnival of vendors touting AI features and roadmaps.  How many will have metrics to support the quality of their AI-abetted outcomes?  How many have forgotten the comma while chasing the cash? Per Upton Sinclair, ““It is difficult to get a man to understand something, when his salary depends on his not understanding it.”

Unquestionably, we must reduce the cost of discovery to protect the portals of justice.  Justice no one can afford to pursue is no justice at all.  But there are uniquely human characteristics we should continue to esteem in discovery, like curiosity, intuition, suspicion and impression; the “Spidey-sense” we derive from tone, nuance and detail.  Before we use AI to summarize collections then deploy AI to characterize the summaries, can we pause just long enough to see if it’s going to work? Real testing, not just that which supports salaries.

Policy for Student Use of AI

11 Monday Dec 2023

Posted by in Uncategorized

≈ 8 Comments

Tags

, , , , , ,

Andy Williams used to croon that this is “The Most Wonderful Time of the Year.” For me, it’s time to update the curriculum for my class on Electronic Discovery and Digital Evidence at the University of Texas in the graduate schools of Law, Computer Science and Information Science. I’ve long built the course around a Workbook I wrote with readings and some two dozen exercises. But, when I last taught the course a year ago, generative AI was hardly a twinkle in Santa’s eye. Now, of course, AI is the topic that’s eaten all others. So, I’ve had to fashion a policy for student use of AI. I elected to embrace student use of AI tools, in part because legal scholarsip is artful plagarism termed “precedent” and–let’s face it–students are going to use LLMs, whatever I say. So, here’s what I’ve come up with. I’ll be grateful for your feedback as comments, most especially if you are an educator facing the same issues with advice born of experience.

Use of Generative Large Language Models to Assist with Exercises

1. Explicit Disclosure Requirement

  • It is a violation of the honor code to misrepresent work by characterizing it as your own if it is not.  Students may use generative LLMs, such as ChatGPT or Bard, for assistance in completing Workbook exercises; however, they must explicitly disclose the use of these tools by providing a brief note or acknowledgment in their submissions. Transparency is mandatory.

2. Verification and Cross-Checking

  • Students may utilize generative LLMs during Workbook exercises but are required to independently verify and cross-check the information generated by these models through additional research using alternate, reliable sources.

3.  Accountability

  • While generative LLMs are permitted tools, students are held accountable for the accuracy and completeness of the information obtained from these models. Any errors or omissions resulting from the use of LLMs are considered the responsibility of the student. This policy underscores the importance of independent verification and personal accountability.

4. Prohibited for Quizzes and Exams

  • Notwithstanding the foregoing, you may not consult any source of information, including AI resources, when completing quizzes or the final exam.

POSTSCRIPT: I add this a day after the foregoing, after reading that the Fifth Circuit’s proposed a rule change requiring that counsel and pro se litigants certify of any filed document, that “no generative artificial intelligence program was used in drafting the document…or to the extent such a program was used, all generated text, including all citations and legal analysis, has been reviewed for accuracy and approved by a human.” I recall shaking my head at how foolish it was when a grandstanding district court judge made headlines by requiring such certifications following a high-profile gaffe in New York. “Of course a lawyer must verify the accuracy of legal analysis and citations! Lawyers shouldn’t need to certify that we did what we are required to do!”

Yet, here I am requiring my students to do much the same. I feel confident in advising students that, if they use AI, they must verify the information and sink or swim based on what they submit, even if the AI hallucinates or misleads. Back in the day, lawyers knew they had to “Shepardize” citations to verify that the cases cited were still solid. Proffering a a made-up citation was beyond comprehension.

So, am I right to require explicit disclosure of generative AI? Or will AI soon be woven into so many sources of information that disclosure will feel as foolish as requiring students to disclose they used a word processor instead of a typewriter would have been forty years ago? I’m struggling with this. What do you think?