by David Yokum, Christopher Robertson and Bernard Chao
Vol. 102 No. 3 (2018) | Crowdsourcing and Data Analytics | Download PDF Version of Article
By protecting the right to a jury, the state and federal constitutions recognize the fundamental value of having civil and criminal disputes resolved by laypersons. Actual trials, however, are relatively rare, in part because parties seek to avoid the risks and cost of trials, and courts seek to clear dockets efficiently. But as desirable as settlement may be, it can be a difficult way to resolve a dispute. Parties view their cases from different perspectives, and these perspectives often cause both sides to be overly optimistic and to expect unreasonably large or unreasonably small resolutions.
This article describes a novel method of incorporating layperson perspectives to provide parties with more accurate information about the value of a case: We suggest that parties work with mediators or settlement judges to create mini-trials and recruit hundreds of online mock jurors to render decisions. By applying modern statistical techniques to these results, the mediators could show the parties the likelihood of possible outcomes and collect qualitative information about strengths and weaknesses for each side. These data will better inform the parties’ views and should thereby facilitate settlement.
Most cases settle. One study examining two federal district courts found an average settlement rate of 70 percent, with large variations based on case type and venue.1 Unfortunately, the settlement process is not easy, and settlement often does not occur until late in discovery or even until the eve of trial.2 Parties frequently spend a great deal of time and money before reaching that point. Earlier settlements would allow parties to save significant resources and would lessen the burden on the court system.
To avoid these costs, courts often push parties to settle. Courts use two techniques to promote settlement. First, courts design discovery rules to force opposing parties to disclose both their arguments and supporting evidence early. This prevents trial by ambush and — most importantly for our purposes — empowers the parties to more accurately predict the eventual outcome of their cases. Furnished with this information, the parties are more likely to settle.3 After all, why spend money going to trial when both parties can already predict the likely outcome? Second, courts regularly suggest, or even require, that the parties participate in some form of alternative dispute resolution (ADR). Mediation is the most popular form of ADR.4 It is a consensual, confidential, and problem-solving process that is intended to facilitate settlement. In 2012, the Federal Judicial Center reported that 63 of 94 district courts authorized judges to require mediation.5 But mediation is not the only form of ADR intended to facilitate settlement. Early Neutral Evaluation (ENE) is another, albeit less popular, type of ADR. ENE uses subject-matter experts to provide the parties with a second opinion on the merits of the case. Twenty-three district courts authorize the use of some form of ENE.6
Despite liberal discovery rules and the help of skilled professionals, opposing parties typically have very different views on the relative value of their cases. The existing types of ADR cannot always bridge that gap. There are both cognitive and practical reasons why this is the case. First, studies have demonstrated that lawyers have overly optimistic views of case outcomes.7 One study asked 481 litigators from 44 different states questions about one of their pending cases that was expected to go to trial in the next 6-to-12 months.8 The litigators were asked: “What would be a win situation in terms of your minimum goal for the outcome of this case?” The attorneys were then asked to estimate the likelihood that they would achieve that goal or better.9 The median attorney thought there was a 70 percent likelihood of achieving the goal. But when the same attorneys were interviewed after their cases ended (in settlement or trial), only 56 percent of the attorneys had achieved their minimum goal.10 This suggests the litigators were overly optimistic about their case outcomes.11
It is not just attorneys that overvalue their cases. Studies suggest that self-interest skews opposing parties’ evaluations of case value. For example, one study randomly assigned 160 students to negotiate on behalf of either the plaintiff or the defendant in a mock case.12 The case involved an injured motorcyclist that was suing an automobile driver for $100,000. After reading identical sets of facts, but before negotiating, the students were asked to predict what a judge would award in the case and what a fair settlement would be from the vantage point of a neutral third party.13 On average, a plaintiff’s prediction of the judge’s award was $14,527 higher than a defendant’s prediction ($38,953 vs. $24,426), and a plaintiff’s estimate of a fair settlement was $17,709 higher than a defendant’s ($37,028 vs. $19,318).14 These results suggest that different sides may not be able to agree on the likelihood of winning or losing nor the potential damages if the plaintiff prevails. Studies comparing settlement offers to outcomes bear out this problem. Parties often go to trial and obtain an outcome that is either only equal or inferior to what they could have achieved from settlement.15
One might expect that mediation or ENE could counteract each party’s optimism bias by giving the parties a realistic view of the likely outcome. But these tactics can only do so much to eliminate bias. First, while mediators often highlight the weaknesses of the parties’ arguments, they rarely give their opinion on the expected outcome. Indeed, nearly half of attorneys do not think it is appropriate for mediators to recommend a particular settlement.16 Even when a mediator does evaluate a case, parties may not trust the evaluation. They understand the mediator’s primary goal is to settle the case and often believe “that the mediator will manipulate how he or she behaves and what he or she says” accordingly.17 Second, the neutral evaluator is only one person, albeit usually a very experienced attorney. But no matter how smart or experienced, one attorney cannot accurately predict how a jury will decide. In some ways, the neutral observer’s training and experience may actually make it more difficult to imagine the case from the perspective of a juror.
That is why parties have traditionally used mock juries when they want to understand their own cases. But mock juries have weaknesses, too. Jury consultants typically recruit 12, 24, or 36 mock jurors. Those numbers are simply not enough to make a prediction with any precision. One or two individuals with extreme views can dramatically change the apparent value of the case. That is why Gallup polls never survey so few people; a larger sample size is needed to provide estimates with a reasonable margin of error. More than one third of the federal caseload is in multi-district litigation (MDL), where hundreds or thousands of cases raise similar issues, potentially implicating hundreds of millions, or billions, of dollars in liability. Typically, to help resolve these cases en masse, judges hold a few “bellwether” trials in front of real juries. While these data-points are better than nothing, a dozen individual jurors, or even 144 jurors across a dozen such trials, are hardly enough to provide a reliable estimate of the total liability exposure.
Crowdsourcing can provide larger and more reliable samples. Crowdsourcing is the practice of using large numbers of people over the internet to perform a task or set of tasks. A variety of platforms recruit such participants for relatively small fees.18 One task these participants can perform is serving as mock jurors. In using hundreds of mock jurors, a basic statistical analysis can provide a far more precise estimate of case outcomes than litigants have been able to generate via other means. Moreover, while it makes sense for parties to be skeptical of a single neutral’s case evaluation, it is more difficult to doubt the collective opinion of hundreds of mock jurors. With the emergence of online crowdsourcing, third-party neutrals can obtain this information relatively quickly and inexpensively. As this practice becomes more popular, we hope to replace online crowdsourcing platforms with prospective jurors that are part of the venire. Currently, most of these individuals typically sit around a jury waiting room without performing any meaningful civic service. A more efficient use of their time would be serving as mock jurors. Eventually, these individuals may be able to work remotely, making service more convenient and less time consuming.
We propose that either a neutral, mediator, or even the parties alone, can themselves conduct a mini-trial using vignettes that represent the particular facts and law of the case. As in a real trial, mock jurors can render a verdict deciding liability and damages.
This is how it works. The attorneys from the parties create shortened presentations of their case, which social scientists call the “stimulus.” This is essentially a version of what the attorneys provide to a neutral during a settlement conference. Depending on how much time and money the parties are prepared to spend, there are two primary options. The more elaborate version involves making a video. In the video, each side offers a presentation with narration read by one of their attorneys.19 Just like a presentation that an attorney might give to a neutral, the presentation can include photographs, documents, animations, or even videotaped deposition testimony. This sort of audio-visual presentation has the advantage of being an immersive experience for the jurors. We have found that the core of a case, even if relatively sophisticated, can often be presented in about 15 minutes per side.20 For smaller-stakes cases, the parties do not have to make videos. They can simply draft a short statement of their position. Similar to the video, the statement can incorporate important evidence. The jurors then read the text and review the excerpts of evidence. To the extent that there are evidentiary objections, the neutral can rule on them; any objectionable evidence and arguments can be excluded from the parties’ presentations. If the neutral is unsure about how the judge will later rule on a key admissibility question, the case can be tried both ways. When the parties’ presentations are ready, the neutral combines the different sides’ presentations and adds jury instructions.
Next, mock jurors are recruited from one of several crowdsourcing platforms. The panel should be broadly representative of the population of potential jurors. The parties should offer incentives to ensure that jurors pay attention. The cost will vary depending on the length of the mini-trial and the number of mock jurors. The neutral can adjust both variables based on the complexity and value of the case. To take advantage of modern statistical techniques, we generally recommend several hundred mock jurors.21 But if the parties simply want to get a sense of what a jury might decide, they can elect for a smaller number. Regardless of how many jurors are chosen, the cost will likely be far less expensive than traditional mock juries. Participants on crowdsourcing platforms are typically paid minimum wage or even less.22 When the trial only lasts half an hour, the cost for each mock juror will be minimal.
Online mock trials can be completed quickly; the authors have run cases in less than a week and sometimes in a day. The results can be provided in easy-to-digest form. For example, a summary could state that 324 out of 800 jurors, or 40.5 percent, determined that the defendant was liable. Those who did find liability set damages at an average of $1.42 million dollars, with a 95 percent confidence interval ranging from $450,000 to $3.7 million. The wide range of estimated damages may explain why the opposing parties have different views of the potential exposure. A small percentage of mock jurors appear to agree with each side’s original intuition about the outcome. But do the parties want to risk relying on the remote possibility that a trial jury will be made of outliers? The parties will likely feel pressure to reach a settlement amount that is near the case’s average expected value as estimated by a mock jury.
We can also calculate a case’s expected value by combining the data on verdicts and damages: Multiply the plaintiff’s chance of winning by the average recovery when winning. Here, the case expected value is $575,100. Of course, special jury forms can also impose comparative fault, third-party fault, or other affirmative defenses to yield a realistic estimate of case value. This statistical evidence is far more information than either a neutral or traditional mock jury could provide the parties. It is also less biased than information from a neutral. The mock jurors are not trying to convince the parties to settle. They are just asked to render their verdict.
For MDL cases, class actions, or other sorts of mass actions where there are systematic variations in the individual cases, variations can be built into the trial stimulus. For example, suppose that some plaintiffs were exposed to a toxic substance when relatively young, and other individuals were exposed later in life, a difference that could affect causation. A dozen such variations could be built into the case as a true randomized experiment, which would allow the analyst to produce a more accurate overall estimate as to liability and also create estimates for subgroups within the pool, just as a public opinion poll can show overall support for the president and break out results by political affiliation.
Although both neutrals and litigants may be wary of trying this “new” approach to valuing cases, they should take comfort in the fact that the use of vignettes is a well-rooted practice in the social sciences.23 Vignette-based experiments are now published in leading scientific journals to predict real-world behaviors.24 Such studies also are found in a wide range of legal contexts, including international law, torts, criminal procedure, contracts, and securities.25 Researchers have studied these online experiments and concluded that they are both reliable and replicable.26 Admittedly, there is still a need for studies that test how well mock juries watching vignettes correspond to real juries in real trials, as there are some reasons to believe that mock juries’ decisions may differ when there are real consequences to their decisions.27 Social scientists are currently studying these limitations. However, for the present purpose of effectively and inexpensively predicting litigation outcomes, these vignettes are the most promising approach modern science offers.
Online, crowd-sourced mock juries can radically change both how parties settle cases and the rate of such settlements. The method is economical and effective. Attorneys and mediators are unlikely to have the necessary technical skills, but companies are emerging to fulfill the demand.28 Attorneys with social science training may be able to run crowd-sourced mini-trials themselves. This is just the beginning. If crowdsourcing can estimate case values for settlement, it can do so for other purposes as well. Insurance companies can estimate exposures more accurately. When clients and their attorneys cannot decide whether their case should be settled or taken to trial, crowdsourcing can provide the answer. Before long, this approach will be a necessary part of every attorney’s legal toolkit.
Footnotes:
