Client Alert
Proportionality and Technology Assisted Review—The Evolving Post-Amendment Landscape
December 13, 2018
By Kevin Broughel, James Worthington, Thomas Barnett & Alex Ehmke
Introduction
In prior Client Alerts, we have written about the heightened focus on proportionality in discovery flowing from the December 2015 amendments to the Federal Rules of Civil Procedure. At that time, there was considerable speculation about the impact that the Rule amendments would have on the use of technology in defining the scope of discoverable information. Of course, prior to the amendments courts had already begun to acknowledge the potential for increased efficiency and accuracy arising from the use of technology assisted review tools (“TAR”) to identify and produce relevant materials from large sets of electronically stored information. Nevertheless, cases that substantively grappled with the intersection of technology and proportionality were far and few between. That was expected to change with the new Rule amendments.
Three years later, those expectations are only partially satisfied. While, as discussed below, a number of cases have delved into the use of evolving technologies for discovery, in large part the case law is only just beginning to scratch the surface. This does not mean, however, that courts are ignoring the impact of TAR on discovery. Instead, it likely reflects a judicial preference for consensual (as opposed to litigated) resolutions of discovery disputes—a preference which has been reinforced by the heightened focus on relevance and proportionality in amended Rule 26(b), as well as the continued widespread use of older methods like keyword searches even as TAR becomes more common.
Recap of the Rule Amendment
As explained previously, amended Rule 26(b) moves the concept of proportionality from its former position in section (b)(2) (“Limitations on Frequency and Extent”) into section (b)(1) (“Scope in General”). Information is discoverable if it is “relevant to any party’s claim or defense and proportional to the needs of the case, considering the importance of the issues at stake in the action, the amount in controversy, the parties’ relative access to relevant information, the parties’ resources, the importance of the discovery in resolving the issues, and whether the burden or expense of the proposed discovery outweighs its likely benefit.” The old discoverability standard of information being “reasonably calculated to lead to the discovery of admissible evidence” is no longer operative.
The Key Case Law and Themes
Judicial analysis of TAR and management of electronically stored information in discovery began before the December 2015 Rule amendments. For example, in the seminal 2012 case of Da Silva Moore et al. v. Publicis Groupe, former Magistrate Judge Andrew J. Peck endorsed the use of TAR.[1] After Da Silva, several other court decisions followed that endorsement.[2] By March 2015, Judge Peck had declared in a separate decision, Rio Tinto PLC v. Vale S.A.,[3] that the acceptability of TAR had become “black letter law.” With that judicial proclamation on the heels of renewed interest in proportionality with the forthcoming Rule amendments, the invitation for subsequent decisions to address the intersection of TAR and proportionality was clear. Although only a handful of cases have since accepted that invitation, a number of themes have started to emerge.
Reluctance to Order TAR at the Outset of Discovery
In Hyles v. New York City,[4] Magistrate Judge Andrew Peck addressed an issue he explicitly left open in his Rio Tinto opinion: at the outset of discovery, can a requesting party force a producing party to use TAR?[5] The plaintiff in Hyles urged the City to use TAR instead of keyword searches. The City refused, and the plaintiff then asked the court to order the City to use TAR. Despite its recognition that “[t]he Court would have liked the City to use TAR in this case,” in part because it is generally “cheaper, more efficient and superior to keyword searching,” the court held that it had no basis on which to force the City to do so.[6] The court reiterated its belief—reflected in the Sedona Conference Cooperation Proclamation—that responding parties are best situated to assess their methods and means of production, and deferred to the City’s preferred keyword approach. The court also emphasized that even if the plaintiff could prove the objective superiority of TAR in this case, it would not compel a different result. The court framed its analysis around the obligation that discovery efforts be reasonable and proportional, explicitly holding that the relevant standard is not “perfection” and that a party may not be ordered to use the “best tool,” so long as the tools that it does employ are reasonable.
The same issue appeared several months later in In re Viagra Products Liability Litigation.[7] The dispute centered on a defendant who wanted to scope its discovery responses with a keyword list developed through an iterative process of negotiation, sampling and testing, and plaintiffs who sought an order compelling the defendant to use a TAR/predictive coding process with input from representatives of both parties. Noting that “no court has ordered a party to engage in TAR and/or predictive coding over the objection of the party,” the In re Viagra court declined to depart from this precedent, ruling that “[e]ven if predictive coding were a more efficient and better method, which [defendant] disputes, it is not clear on what basis the court could compel [defendant] to use a particular form of ESI, especially in the absence of any evidence that [defendant’s] preferred method would produce, or has produced, insufficient discovery responses.”[8] In re Viagra is thus consistent with a pattern of judicial deference to litigants in choosing their preferred discovery technologies, so long as they satisfy their discovery obligations.
This is not to say, however, that a court will shy away from ordering TAR if it determines that the party’s first applied discovery protocol is insufficient. For instance, in Winfield v. City of New York,[9] the City’s ESI protocol initially consisted of applying a “heavily negotiated” set of search terms, with some court guidance, against a set of custodial data. When plaintiffs objected to the pace of the City’s review, the court then directed the City to begin using TAR to complete discovery for that set of data. Ironically, later on with respect to a different data set, the City agreed to run an expansive keyword list, but only on the condition that it be permitted to use its predictive coding system to first reduce the population of in-scope documents. Plaintiffs objected, arguing that the City’s TAR process to that point had over-designated documents as non-responsive and privileged. Plaintiffs requested that the City be prevented from using TAR further unless the City made substantial disclosures about its TAR processes, including samples of non-responsive documents and explanations of the TAR’s “ranking” system. Based on factors such as the large size of the seed set employed (more than 7,200 documents), the City’s confidential validation process described to the court in camera, and the City’s use of five TAR training rounds, the court held there was nothing “inherently defective” with the City’s TAR process.[10] Nevertheless, the court did require the City to produce to plaintiffs multiple samples of non-responsive and non-privileged documents for plaintiffs’ review, reasoning that it would “increase transparency” and help plaintiffs determine if there were more responsive documents to be produced and more TAR training necessary.[11]
Production Adequacy—Perfection is Not Required
No single discovery method is perfect. Several post-amendment cases challenging TAR protocols underscore that principle and also serve as a reminder that the Rule 26(g) standard of making a “reasonable inquiry” when responding to a discovery request applies regardless of the methods chosen to review and produce responsive materials.
For instance, in a well-publicized, pre-Amendment opinion, the tax court in Dynamo Holdings Limited Partnership v. Commissioner of Internal Revenue[12] approved the use of predictive coding to review and produce data from two backup storage tapes where the producing party claimed it would be overly burdensome and expensive to otherwise comply with the discovery request.
Two years later, the parties were back before the tax court on the Commissioner’s motion to compel production of over 1,350 relevant documents it claimed were not captured through the predictive coding process framed by a prior Boolean search.[13] The responding party argued that the documents that were excluded were properly deemed irrelevant by the predictive coding algorithm.
In examining the predictive coding process, the tax court noted that the parties had followed an agreed framework. That agreement included the Commissioner providing its own list of search terms for the responding party to initially run against the data, reviewing seed sets where the Commissioner identified relevant and irrelevant documents to “train” the predictive coding model to identify responsive documents, and allowing the Commissioner to establish the recall rate, here 95%, for the predictive coding process. From this, the tax court reasoned that even if the predictive coding response was flawed, no relief was warranted. Among other reasons, the tax court observed that the Tax Court Rules, as well as Rule 26(g), only required a party to make a “reasonable inquiry” when responding to discovery regardless of whether the responding party used TAR, keywords, or manual review.[14] Here, the tax court held, that standard was satisfied by the predictive coding protocol developed by the responding party with the Commissioner’s input, even with its perceived shortcomings.
The more recent case of In re Broiler Chicken Antitrust Litigation[15] is perhaps the most detailed published order addressing the specifics of ESI search and production adequacy. Although the order contains numerous provisions that may be unnecessary in any particular case, it is a useful reference guide which may help litigants assess whether a TAR-driven production is satisfactory under Rule 26(g). The order addresses—and defines potential solutions to—numerous issues that routinely arise in the course of discovery negotiations, including: (i) disclosure of collection and culling parameters, (ii) disclosure of proposed TAR procedures, (iii) TAR meet-and-confer and dispute resolution procedures, (iv) iterative search term negotiation and testing procedures, and (v) a validation protocol outlining “quality control and quality assurance procedures to ensure a reasonable production consistent with the requirements of Federal Rule of Civil Procedure 26(g),” whether a party employed TAR or manual review.
The Order’s discussion of “adequacy” in the context of a review may be of particular interest:
It should be noted that, when conducted by [a subject matter expert], a recall estimate on the order of 70% to 80% is consistent with, but not the sole indicator of, an adequate (i.e., high-quality) review. A recall estimate somewhat lower than this does not necessarily indicate that a review is inadequate, nor does a recall in this range or higher necessarily indicate that a review is adequate; the final determination also will depend on the quantity and nature of the documents that were missed by the review process.[16]
“Adequacy” is, of course, an inherently case-specific analysis, but it will be interesting to see if such framings of the issue in the terms of contemporary e-discovery concepts like recall percentages gain broader traction with judges faced with the problem of adjudicating the proper—and proportionate —boundaries of discovery.
Transparency and Validation Are Becoming More Prominent
Although, as discussed above, courts generally remain reluctant to compel unwilling litigants to employ TAR in the first instance, in a growing number of recent cases courts have demonstrated a willingness to engage with the technical details of TAR processes in the course of adjudicating discovery disputes. The case of City of Rockford v. Mallinckrodt[17] is a very recent example of a court engaging substantively with validation procedures in the context of a discovery dispute.
In Rockford, the court praised the parties for their extensive cooperation in negotiating a keyword searching protocol, with the option to also employ TAR, for what was expected to be a universe of millions of electronic documents. The sole issue before the court was how to validate the parties’ ESI production procedures after the fact. While defendants advocated an ongoing dialogue between the parties about any documents they reasonably believed were wrongly withheld, plaintiffs urged a statistical approach, requesting that the parties be directed to generate samples from their respective “null sets”—the populations of documents deemed non-responsive—and assess their respective rates of error (i.e., the proportion of documents marked non-responsive that were, in fact, responsive).
The court granted plaintiffs’ request, finding that the sampling proposal was both reasonable and proportionate. Stressing that “[v]alidation and quality assurance are fundamental principles to ESI production,” the court deemed the proposal reasonable because it would satisfy these goals (analogizing the after-the-fact review to that performed in the typical TAR process).[18] With respect to proportionality, the court generally dismissed the defendants’ concerns regarding the cost and burden of the sampling procedure as unfounded. It also noted numerous countervailing interests that justified the expense, including the media attention garnered by the case, the amount at issue, and the defendants’ resources. Given this weight of the interests, the court ordered the creation, review and analysis of the null sets.
Keywords Searches Remain a Common ESI Review Protocol
For better or for worse, the post-2015 Rule amendment cases reveal that keyword searches are still common. As the Rockford court observed, “[E]ven as TAR becomes more acceptable and understood, in certain circumstances, key word searching and even linear review are not necessarily unreasonable…Indeed, when being pitched on the virtues of TAR, some parties, attorneys and courts may feel – albeit without any evidence-based reason – as though they are being sold a monorail.”[19] There are a number of cases where courts are presented with ESI scope disputes over keywords. In these cases, courts generally will either order certain keyword searches to be performed or that search terms be modified by applying proportionality principles.[20]
Conclusion
In sum, the discovery landscape—driven by the ongoing explosion in volumes of party data—remains in a state of flux. While the use of TAR continues to expand, the number of decisions addressing TAR in the context of proportionality has grown at a relatively slow pace. This is presumably in large part a reflection of courts continuing to adhere to the principle that they “should generally not play a role in dictating the design of a search, choosing search tools, selecting search terms, or designating custodians, unless a responding party’s choice ‘is manifestly unreasonable or the requesting party demonstrates that the resulting production is deficient.’”[21] However, given the renewed focus on proportionality since the 2015 Rule amendments, and the increasing sophistication of tools allowing parties—and, in turn, judges—to quantify the reasonableness of litigants’ discovery efforts, one would expect to see more decisions in the coming year utilizing TAR measures and concepts in framing the proportionality analysis. This trend seems likely only to accelerate, as innovation born out of necessity continues to characterize large-scale discovery practice.
[1] 287 F.R.D. 182 (S.D.N.Y. 2012), adopted sub nom. Moore v. Publicis Groupe SA, No. 11 CIV. 1279 ALC AJP, 2012 WL 1446534 (S.D.N.Y. Apr. 26, 2012).
[2] See, e.g., Hinterberger v. Catholic Health Sys., Inc., No. 08-CV-380S F, 2013 WL 2250603, at *1 (W.D.N.Y. May 21, 2013) (reflecting the court’s “dissatisfaction with the parties’ lack of progress” toward resolving discovery issues and “point[ing] to the availability of predictive coding . . . [and] directing the parties’ attention to” the Da Silva decision); F.D.I.C. v. Bowden, No. CV413-245, 2014 WL 2548137, at *13 (S.D. Ga. June 6, 2014) (ordering that “the parties shall consider the use of predictive coding”); Chevron Corp. v. Donziger, No. 11 CIV. 0691 LAK, 2013 WL 1087236, at *32 (S.D.N.Y. Mar. 15, 2013) (noting the court’s prior request that the parties consider using predictive coding to “reduce the burden and effort” of discovery).
[3] 306 F.R.D. 125, 127 (S.D.N.Y. 2015).
[4] No. 10CIV3119ATAJP, 2016 WL 4077114 (S.D.N.Y. Aug. 1, 2016).
[5] Rio Tinto, 306 F.R.D. at 127 n.1.
[6] Hyles, 2016 WL, at *2–3.
[7] No. 16-MD-02691-RS (SK), 2016 WL 7336411 (N.D. Cal. Oct. 14, 2016).
[8] Id. at *2.
[9] No. 15CV05236LTSKHP, 2017 WL 5664852 (S.D.N.Y. Nov. 27, 2017).
[10] Id. at *11.
[11] Id. at *11.
[12] 143 T.C. 183 (2014).
[13] 2016 WL 4204067 (T.C. July 13, 2016).
[14] In particular, the tax court quoted the following from Magistrate Judge Peck in Rio Tinto: “One point must be stressed—it is inappropriate to hold TAR [technology assisted review] to a higher standard than keywords or manual review. Doing so discourages parties from using TAR for fear of spending more in motion practice than the savings from using TAR for review.” Rio Tinto, 306 F.R.D. at 129.
[15] No. 1:16-CV-08637, 2018 WL 1146371 (N.D. Ill. Jan. 3, 2018).
[16] Id. at *6.
[17] 326 F.R.D. 489 (N.D. Ill. 2018).
[18] Id. at 494.
[19] Id. at 493.
[20] See, e.g., Cen Com Inc. V. Numerex Corp., 2018 WL 1737943 (W.D. Was. April 11, 2018) (compelling plaintiff to run search terms); Abbott v. Wyoming County Sheriff’s Office, 2017 WL 2115381 (W.D.N.Y. May 16, 2017) (ordering defendant to search e-mails that contained plaintiff’s name or other names contained in plaintiff’s discovery response, as well as ten additional search terms); Hall v. Rent-A-Ctr., Inc., No. 4:16CV978, 2018 WL 4293141, at *6–8 (E.D. Tex. Aug. 31, 2018) (denying plaintiffs’ request for 13 specific keyword searches, granting as to three others, modifying the specific terms of one in the interest of proportionality, and deferring judgment on 46 remaining searches).
[21] Commins v. NES Rentals Holdings, Inc., 2018 WL 3186983, at *3 (W.D. Ky. Jun. 28, 2018) (quoting Mortgage Resolution Servicing, LLC v. JPMorgan Chase Bank, N.A., 15-CV-0293, 2017 WL 2305398, at *2 (S.D.N.Y. May 18, 2017)).