Today, Doug Austin‘s splendid eDiscoveryToday blog featured O’Donnell/Salvatori Inc. v. Microsoft Corp., No. C20-882-MLP (W.D. Wash. Oct. 1, 2021), where a U.S. Magistrate sitting in Seattle opined on an issue I wrote about nine years ago (when there wasn’t a case to be found on the question): “Must a party produce all ESI retrieved from the use of negotiated search terms?” Magistrate Peterson wisely held that “a party’s agreement to run search terms does not waive its right to review the resulting documents for relevance so long as the review can be done in a reasonably timely manner.”
Because the issue remains contentious, I thought reprinting my long ago post and the associated practice tip (that would have kept the parties from tripping up) might be timely. Here it is (from March 22, 2013):
More than once, I’ve faced disputes stemming from diametrically different expectations concerning the use of keywords as a means to identify responsive ESI. I don’t recall seeing a case on this; but, it wouldn’t surprise me if there was one. If not, there soon will be because the issue is more common than one might imagine.
When requesting parties hammer out agreements on search terms to be run against the producing party’s ESI, sometimes the requesting party’s expectation is that any item responsive to the agreed-upon keywords (that is, any item that’s “hit”) must be produced unless withheld as privileged. Put another way, the requesting party believes that, by agreeing to the use of a set of keywords as a proxy for attorney review of the entire potentially-responsive collection, and thereby relieving the producing party of the broader obligation to look at everything that may be responsive, those keywords define responsiveness per se, requiring production if not privileged.
Now I appreciate that some are reading that and getting hot under the collar. You’re saying things like:
- “We always have the right to review items hit for responsiveness!”
- “It’s the Request for Production not the keyword hits that define the scope of e-discovery!”
- “Nothing in the Rules or the law obliges a party to produce non-responsive items!”
Perhaps; but, there’s sufficient ambiguity surrounding the issue to prompt prudent counsel to address the point explicitly when negotiating keyword search protocols, and especially when drafting agreed orders memorializing search protocols.
To appreciate why expectations should be plainly stated, one need only look at the differing incentives that may prompt disparate expectations.
What is a producing party’s incentive to limit the scope of search to only a handful of queries and keywords? Federal law requires a producing party to search all reasonably accessible sources of information that may hold responsive information and to identify those potentially responsive sources that won’t be searched. That’s a pretty broad mandate; so, it’s no wonder producing parties seek to narrow the scope by securing agreements to use keyword queries. Producing parties have tons of incentive to limit the scope of review to only items with keyword hits. It eases their burden, trims their cost and affords requesting parties cover from later complaints about scope and methodology.
What is the requesting party’s incentive to limit an opponent’s scope of search to only those items with keyword hits? Requesting parties might respond that their incentive is to insure that they get to see the items with hits so long as they are not privileged. By swapping keyword culling for human review, requesting parties need not rely upon an untrusted opponent’s self-interested assessment of the material. Instead, if it’s hit by the agreed-upon keywords, the item will be produced unless it’s claimed to be privileged; in which case the requesting party gets to see its privilege log entry. That’s often the contemplated quid pro quo.
Both arguments have considerable merit; and, yes, you can be compelled to produce non-responsive items, if the agreement entered into between the parties is construed to create that obligation. Some might argue that the agreement to use queries is an agreement to treat those queries as requests for production. You don’t have to agree, dear reader; but, you’d be wise to plan for opponents (and judges) who think this way.
These are issues we need to pay attention to as we move closer to broader adoption of technology-assisted review. We may be gravitating to a place where counsel’s countermanding a machine’s “objective” characterization of a document as responsive will be viewed with suspicion. Responding parties see electronic culling as just an extension of counsel’s judgment; but, requesting parties often see electronic culling as an objective arbiter of responsiveness. Face it: requesting parties believe that opponents hide documents. TAR and keyword search may be embraced by requesting parties as a means to get hold of helpful documents that would not otherwise see the light of day.
Practice Tip: If you enter into an agreement with the other side to use keywords and queries for search, be clear about expectations with respect to the disposition of items hit by queries. Assuming the items aren’t privileged, are they deemed responsive because they met the criteria used for search or is the producing party permitted or obliged to further cull for responsiveness based on the operative Requests for Production? You may think this is clear to the other side; but, don’t count on it. Likewise, don’t assume the Court shares your interpretation of the protocol. Just settling upon an agreed-upon list of queries may not be sufficient to insure a meeting of the minds.
William Hamilton said:
Isn’t the background issue here that too many attorneys do not understand that keyword searching is only looking for words, or in IT speak “tokens,” that serve as predictive markers. The better the keyword search the more likely the “hit” document is relevant, but its all a game of prediction that requires subsequent human review to identify which of the predictions are accurate.
Agree as to the predictive usage; but whether the keywords are “better” or “worse” doesn’t resolve the issue as to whether or not subsequent human review may be undertaken. Arguably, a set of hard-fought and court-blessed agreed queries are “ideal by consensus” (to lawyers, if not to information scientists).
Privilege aside, why should a human get to override a carefully-crafted negotiated process? After all, humans haven’t shown themselves to be especially gifted predictors of relevance (based on longstanding studies of human review), and there’s no data I’ve seen that proves that human linear review alone is superior to keyword culling alone. My expectation is that mistaken assumptions and attention fatigue would relegate humans to a distant third place compared to other search mechanisms.
I think the decision a sound one, but the better approach would have been better communication and explicit agreement between the parties at the outset.
Doug Austin said:
Thanks for covering my blog post, Craig! Doesn’t surprise me at all that you provided great advice on this topic BEFORE all the cases that were referenced in the decision — which I referenced when I discussed this case in the ACEDS #caseoftheweek earlier today.
Speaking of which, one of the cases referenced in this case was Ball v. Manalto, Inc. and the name of the plaintiff was “Craig Ball”. Small world! 🙂
That’s just creepy that the plaintiff has the same name (different middle initial).
Jason Fulton said:
It seems like an endless merry go round. A receiving party might rightfully want to know what keywords the nonresponsive document contained, leading to production of a “nonresponsive log” and then what, in camera or special master review? If non-responsive but identified by agreed keyword doesn’t disclose something proprietary it would seem better to err on the side of production or some other side process to address those documents.
Either way, its excellent advice to add this agreement up front.
Pingback: Week 44 – 2021 – This Week In 4n6