| , ,

Crowdsourcing and Data Analytics: The New Settlement Tools

by , and

Vol. 102 No. 3 (2018) | Crowdsourcing and Data Analytics | Download PDF Version of Article
A crowd of people

By protecting the right to a jury, the state and federal constitutions recognize the fundamental value of having civil and criminal disputes resolved by laypersons. Actual trials, however, are relatively rare, in part because parties seek to avoid the risks and cost of trials, and courts seek to clear dockets efficiently. But as desirable as settlement may be, it can be a difficult way to resolve a dispute. Parties view their cases from different perspectives, and these perspectives often cause both sides to be overly optimistic and to expect unreasonably large or unreasonably small resolutions.

This article describes a novel method of incorporating layperson perspectives to provide parties with more accurate information about the value of a case: We suggest that parties work with mediators or settlement judges to create mini-trials and recruit hundreds of online mock jurors to render decisions. By applying modern statistical techniques to these results, the mediators could show the parties the likelihood of possible outcomes and collect qualitative information about strengths and weaknesses for each side. These data will better inform the parties’ views and should thereby facilitate settlement.

Settling Cases

Most cases settle. One study examining two federal district courts found an average settlement rate of 70 percent, with large variations based on case type and venue.1 Unfortunately, the settlement process is not easy, and settlement often does not occur until late in discovery or even until the eve of trial.2 Parties frequently spend a great deal of time and money before reaching that point. Earlier settlements would allow parties to save significant resources and would lessen the burden on the court system.

To avoid these costs, courts often push parties to settle. Courts use two techniques to promote settlement. First, courts design discovery rules to force opposing parties to disclose both their arguments and supporting evidence early. This prevents trial by ambush and — most importantly for our purposes — empowers the parties to more accurately predict the eventual outcome of their cases. Furnished with this information, the parties are more likely to settle.3 After all, why spend money going to trial when both parties can already predict the likely outcome? Second, courts regularly suggest, or even require, that the parties participate in some form of alternative dispute resolution (ADR). Mediation is the most popular form of ADR.4 It is a consensual, confidential, and problem-solving process that is intended to facilitate settlement. In 2012, the Federal Judicial Center reported that 63 of 94 district courts authorized judges to require mediation.5 But mediation is not the only form of ADR intended to facilitate settlement. Early Neutral Evaluation (ENE) is another, albeit less popular, type of ADR. ENE uses subject-matter experts to provide the parties with a second opinion on the merits of the case. Twenty-three district courts authorize the use of some form of ENE.6

Despite liberal discovery rules and the help of skilled professionals, opposing parties typically have very different views on the relative value of their cases. The existing types of ADR cannot always bridge that gap. There are both cognitive and practical reasons why this is the case. First, studies have demonstrated that lawyers have overly optimistic views of case outcomes.7 One study asked 481 litigators from 44 different states questions about one of their pending cases that was expected to go to trial in the next 6-to-12 months.8 The litigators were asked: “What would be a win situation in terms of your minimum goal for the outcome of this case?” The attorneys were then asked to estimate the likelihood that they would achieve that goal or better.9 The median attorney thought there was a 70 percent likelihood of achieving the goal. But when the same attorneys were interviewed after their cases ended (in settlement or trial), only 56 percent of the attorneys had achieved their minimum goal.10 This suggests the litigators were overly optimistic about their case outcomes.11

It is not just attorneys that overvalue their cases. Studies suggest that self-interest skews opposing parties’ evaluations of case value. For example, one study randomly assigned 160 students to negotiate on behalf of either the plaintiff or the defendant in a mock case.12 The case involved an injured motorcyclist that was suing an automobile driver for $100,000. After reading identical sets of facts, but before negotiating, the students were asked to predict what a judge would award in the case and what a fair settlement would be from the vantage point of a neutral third party.13 On average, a plaintiff’s prediction of the judge’s award was $14,527 higher than a defendant’s prediction ($38,953 vs. $24,426), and a plaintiff’s estimate of a fair settlement was $17,709 higher than a defendant’s ($37,028 vs. $19,318).14 These results suggest that different sides may not be able to agree on the likelihood of winning or losing nor the potential damages if the plaintiff prevails. Studies comparing settlement offers to outcomes bear out this problem. Parties often go to trial and obtain an outcome that is either only equal or inferior to what they could have achieved from settlement.15

One might expect that mediation or ENE could counteract each party’s optimism bias by giving the parties a realistic view of the likely outcome. But these tactics can only do so much to eliminate bias. First, while mediators often highlight the weaknesses of the parties’ arguments, they rarely give their opinion on the expected outcome. Indeed, nearly half of attorneys do not think it is appropriate for mediators to recommend a particular settlement.16 Even when a mediator does evaluate a case, parties may not trust the evaluation. They understand the mediator’s primary goal is to settle the case and often believe “that the mediator will manipulate how he or she behaves and what he or she says” accordingly.17 Second, the neutral evaluator is only one person, albeit usually a very experienced attorney. But no matter how smart or experienced, one attorney cannot accurately predict how a jury will decide. In some ways, the neutral observer’s training and experience may actually make it more difficult to imagine the case from the perspective of a juror.

That is why parties have traditionally used mock juries when they want to understand their own cases. But mock juries have weaknesses, too. Jury consultants typically recruit 12, 24, or 36 mock jurors. Those numbers are simply not enough to make a prediction with any precision. One or two individuals with extreme views can dramatically change the apparent value of the case. That is why Gallup polls never survey so few people; a larger sample size is needed to provide estimates with a reasonable margin of error. More than one third of the federal caseload is in multi-district litigation (MDL), where hundreds or thousands of cases raise similar issues, potentially implicating hundreds of millions, or billions, of dollars in liability. Typically, to help resolve these cases en masse, judges hold a few “bellwether” trials in front of real juries. While these data-points are better than nothing, a dozen individual jurors, or even 144 jurors across a dozen such trials, are hardly enough to provide a reliable estimate of the total liability exposure.

Crowdsourcing can provide larger and more reliable samples. Crowdsourcing is the practice of using large numbers of people over the internet to perform a task or set of tasks. A variety of platforms recruit such participants for relatively small fees.18 One task these participants can perform is serving as mock jurors. In using hundreds of mock jurors, a basic statistical analysis can provide a far more precise estimate of case outcomes than litigants have been able to generate via other means. Moreover, while it makes sense for parties to be skeptical of a single neutral’s case evaluation, it is more difficult to doubt the collective opinion of hundreds of mock jurors. With the emergence of online crowdsourcing, third-party neutrals can obtain this information relatively quickly and inexpensively. As this practice becomes more popular, we hope to replace online crowdsourcing platforms with prospective jurors that are part of the venire. Currently, most of these individuals typically sit around a jury waiting room without performing any meaningful civic service. A more efficient use of their time would be serving as mock jurors. Eventually, these individuals may be able to work remotely, making service more convenient and less time consuming.

We propose that either a neutral, mediator, or even the parties alone, can themselves conduct a mini-trial using vignettes that represent the particular facts and law of the case. As in a real trial, mock jurors can render a verdict deciding liability and damages.

The Online Trial

This is how it works. The attorneys from the parties create shortened presentations of their case, which social scientists call the “stimulus.” This is essentially a version of what the attorneys provide to a neutral during a settlement conference. Depending on how much time and money the parties are prepared to spend, there are two primary options. The more elaborate version involves making a video. In the video, each side offers a presentation with narration read by one of their attorneys.19 Just like a presentation that an attorney might give to a neutral, the presentation can include photographs, documents, animations, or even videotaped deposition testimony. This sort of audio-visual presentation has the advantage of being an immersive experience for the jurors. We have found that the core of a case, even if relatively sophisticated, can often be presented in about 15 minutes per side.20 For smaller-stakes cases, the parties do not have to make videos. They can simply draft a short statement of their position. Similar to the video, the statement can incorporate important evidence. The jurors then read the text and review the excerpts of evidence. To the extent that there are evidentiary objections, the neutral can rule on them; any objectionable evidence and arguments can be excluded from the parties’ presentations. If the neutral is unsure about how the judge will later rule on a key admissibility question, the case can be tried both ways. When the parties’ presentations are ready, the neutral combines the different sides’ presentations and adds jury instructions.

Next, mock jurors are recruited from one of several crowdsourcing platforms. The panel should be broadly representative of the population of potential jurors. The parties should offer incentives to ensure that jurors pay attention. The cost will vary depending on the length of the mini-trial and the number of mock jurors. The neutral can adjust both variables based on the complexity and value of the case. To take advantage of modern statistical techniques, we generally recommend several hundred mock jurors.21 But if the parties simply want to get a sense of what a jury might decide, they can elect for a smaller number. Regardless of how many jurors are chosen, the cost will likely be far less expensive than traditional mock juries. Participants on crowdsourcing platforms are typically paid minimum wage or even less.22 When the trial only lasts half an hour, the cost for each mock juror will be minimal.

Online mock trials can be completed quickly; the authors have run cases in less than a week and sometimes in a day. The results can be provided in easy-to-digest form. For example, a summary could state that 324 out of 800 jurors, or 40.5 percent, determined that the defendant was liable. Those who did find liability set damages at an average of $1.42 million dollars, with a 95 percent confidence interval ranging from $450,000 to $3.7 million. The wide range of estimated damages may explain why the opposing parties have different views of the potential exposure. A small percentage of mock jurors appear to agree with each side’s original intuition about the outcome. But do the parties want to risk relying on the remote possibility that a trial jury will be made of outliers? The parties will likely feel pressure to reach a settlement amount that is near the case’s average expected value as estimated by a mock jury.

We can also calculate a case’s expected value by combining the data on verdicts and damages: Multiply the plaintiff’s chance of winning by the average recovery when winning. Here, the case expected value is $575,100. Of course, special jury forms can also impose comparative fault, third-party fault, or other affirmative defenses to yield a realistic estimate of case value. This statistical evidence is far more information than either a neutral or traditional mock jury could provide the parties. It is also less biased than information from a neutral. The mock jurors are not trying to convince the parties to settle. They are just asked to render their verdict.

For MDL cases, class actions, or other sorts of mass actions where there are systematic variations in the individual cases, variations can be built into the trial stimulus. For example, suppose that some plaintiffs were exposed to a toxic substance when relatively young, and other individuals were exposed later in life, a difference that could affect causation. A dozen such variations could be built into the case as a true randomized experiment, which would allow the analyst to produce a more accurate overall estimate as to liability and also create estimates for subgroups within the pool, just as a public opinion poll can show overall support for the president and break out results by political affiliation.


Although both neutrals and litigants may be wary of trying this “new” approach to valuing cases, they should take comfort in the fact that the use of vignettes is a well-rooted practice in the social sciences.23 Vignette-based experiments are now published in leading scientific journals to predict real-world behaviors.24 Such studies also are found in a wide range of legal contexts, including international law, torts, criminal procedure, contracts, and securities.25 Researchers have studied these online experiments and concluded that they are both reliable and replicable.26 Admittedly, there is still a need for studies that test how well mock juries watching vignettes correspond to real juries in real trials, as there are some reasons to believe that mock juries’ decisions may differ when there are real consequences to their decisions.27 Social scientists are currently studying these limitations. However, for the present purpose of effectively and inexpensively predicting litigation outcomes, these vignettes are the most promising approach modern science offers.

Online, crowd-sourced mock juries can radically change both how parties settle cases and the rate of such settlements. The method is economical and effective. Attorneys and mediators are unlikely to have the necessary technical skills, but companies are emerging to fulfill the demand.28 Attorneys with social science training may be able to run crowd-sourced mini-trials themselves. This is just the beginning. If crowdsourcing can estimate case values for settlement, it can do so for other purposes as well. Insurance companies can estimate exposures more accurately. When clients and their attorneys cannot decide whether their case should be settled or taken to trial, crowdsourcing can provide the answer. Before long, this approach will be a necessary part of every attorney’s legal toolkit.


  1. Theodore Eisenberg & Charlotte Lanvers, What is the Settlement Rate and Why Should We Care?, J. Empirical Legal Stud. 111, 132 (2009) (counting all cases that were neither dismissed, summarily adjudicated, or tried to verdict). See also Marc Galanter & Mia Cahill, “Most Cases Settle”: Judicial Promotion and Regulation of Settlements, 46 Stan. L. Rev. 1339, 1339–40 (1994) (“Oft-cited figures estimating settlement rates of between 85 and 95 percent are misleading; those figures represent all civil trials that do not go to trial.”(footnote omitted)).
  2. Joel L. Schrag, Managerial Judges: An Economic Analysis of the Judicial Management of Legal Discovery, 30 Rand J. Econ. 305, 308 (1999) (“[M]any settlements are reached ‘on the courthouse steps,’ after discovery costs are sunk.” (footnote omitted)).
  3. See, e.g., Richard A. Posner, An Economic Approach to Legal Procedure and Judicial Administration, 2 J. Legal Stud. 399, 426 (1973) (“A discovery provision that enables both parties to improve their estimates of the outcome of the case is thus likely to facilitate settlement.”).
  4. Jacqueline Nolan-Haley, Mediation: The “New Arbitration”, 17 Harv. Negot. L. Rev. 61, 66-73 (2012).
  5. Alternative Dispute Resolution Now an Established Practice in Federal Courts, United States Courts (June 25, 2012), http://www.uscourts.gov/news/2012/06/25/alternative-dispute-resolution-now-established-practice-federal-courts.
  6. Id.
  7. See, e.g., Elizabeth F. Loftus & Willem A. Wagenaar, Lawyers’ Predictions of Success, 28 Jurimetrics J. 437, 450 (1988) (finding that “lawyers tended to be overconfident, in general, but especially so in cases in which they initially made highly confident predictions.”); Jane Goodman-Delahunty, Pär Anders Granhag, Maria Hartwig & Elizabeth F. Loftus, Insightful or Wishful: Lawyers’ Ability to Predict Case Outcomes, 16 Psychol., Pub. Pol’y, & L. 133, 153 (2010) (“Our large sample of U.S. lawyers showed clear evidence of unrealistic litigation goals. . . .”).
  8. Goodman-Delahunty, supra note 7, at 138.
  9. Id. at 139.
  10. Id. at 140–141.
  11. Others have demonstrated that attorneys suffer from “confirmation bias” and suggested that this also contributes to “an excessively optimist perception of the merits of a case.” See Andrew J. Wistrich & Jeffrey J. Rachlinski, How Lawyers’ Intuitions Prolong Litigation, 86 S. Cal. L. Rev. 571, 602 (2013).
  12. George Loewenstein, Samuel Issacharoff, Colin Camerer & Linda Babcock, Self-Serving Assessments of Fairness and Pretrial Bargaining, 22 J. Legal Stud. 135, 145 (1993).
  13. Id. at 146. The students were given small incentives for accurately predicting the outcome.
  14. Id. at 150–51. Both results were statistically significant at p = .0001 level.
  15. See, e.g., Randall L. Kiser, Martin A. Asher, and Blakeley B. McShane, Let’s Not Make a Deal: An Empirical Study of Decision Making in Unsuccessful Settlement Negotiations, 5 J. Empirical Legal Stud. 551, 566 (2008) (finding that plaintiffs received an award less than or equal to the last defendants’ last offer in 61.2% of the cases while defendants committed a similar error 24.3% of the time). See generally Jeffrey J. Rachlinski, Gains, Losses, and the Psychology of Litigation, 70 S. Cal. L. Rev. 113 (1996).
  16. Task Force on Improving Mediation Quality, Final Report, 2008 A.B.A. Sec. of Disp. Resol. 14.
  17. Wayne D. Brazil, Early Neutral Evaluation or Mediation? When Might ENE Deliver More Value?, 14 Disp. Resol. Mag. 10, 12 (2007).
  18. Krin Irvine, David A. Hoffman & Tess Wilkinson-Ryan, Law and Psychology Grows Up, Goes Online, and Replicates, 15 J. Empirical Legal Stud. 320, 333–35 (2018) (comparing three crowdsourcing platforms).
  19. The authors have created many such vignettes in their academic papers and to advise attorneys in actual cases. See, e.g., Bernard Chao & Roderick O’Dorisio, Testing the White Hat Effect in Patent Law, 27 Fed. Cir. B.J. 155 (2017).
  20. See, e.g., D. Alex Winkelman, David V. Yokum, Lisette C. Cole, Shelby C. Thompson, & Christopher T. Robertson, An Empirical Method For Harmless Error, 46 Ariz. St. L.J. 1405 (2014).
  21. A statistician can estimate the number of mock jurors needed by performing “power calculations” that are based on the expected range of outcomes. For example, damages
    are highly variable, and calculating a reliable estimate often needs far more mock jurors than simply predicting who will win on a single count (a binary variable).
  22. See, e.g., Adam J. Berinksy, Gregory A. Huber & Gabriel S. Lenz, Evaluating Online Labor Markets for Experimental Research: Amazon.com’s Mechanical Turk, 20 Pol. Analysis 351, 353 (2012).
  23. Jessica L. Collett & Ellen Childs, Minding the Gap: Meaning, Affect, and the Potential Shortcomings of Vignettes, 40 Soc. Sci. Res. 513, 513 (2011) (saying that vignettes have become common tools in a range of scientific and practical fields including “sociology, psychology, business, and health sciences”). See generally Rhidian Hughes, Vignette Technique, in The SAGE Encyclopedia of Soc. Sci. Res. Methods 1184, 1184– 85 (Michael S. Lewis-Beck et al. eds., 2004) (discussing the methodology and use of vignette technique).
  24. See, e.g., Aaron S. Kesselhim et al., A Randomized Study of How Physicians Interpret Research Funding Disclosures, 369 New Eng. J. Med. 1119, 1120–21 (2012).
  25. Irvine et al., supra note 18, at 321.
  26. Id. at 344 (“[R]esearchers using random assignment in experiments not otherwise focused on the effects of individual political views have reason for confidence that MTurk, along with similar platforms[,] . . . provides a source of survey respondents which is convenient, cheap, and functionally similar to traditional offline samples.”); Gabriele Paolacci, Jesse Chandler & Panagiotis G. Ipeirotis, Running Experiments on Amazon Mechanical Turk, 5 Judgment & Decision Making 411, 417 (2010) (“Workers in Mechanical Turk exhibit the classic heuristics and biases and pay attention to directions at least as much as subjects from traditional sources.”).
  27. See generally Brian H. Bornstein, The Ecological Validity of Jury Simulations: Is the Jury Still Out?, 23 Law & Hum. Behav. 75 (1999).
  28. The authors have founded one such company. For an example, see Hugo Analytics LLC, http://www.hugoanalytics.com/(last visited Sep. 24, 2018).