Case Management Reform: The Promise of Big Data (Sidebar)

In November 2021, some 30 judges and scholars gathered in Santa Monica, Calif., to discuss the prospects for an emerging era of civil case management reform. The participants included proponents of reform as well as skeptics, but a day of reviewing history, studying examples, and candid give-and-take produced a surprising degree of consensus. Case management reform is both important and feasible, we generally agreed, but it is far more likely to succeed if it has some crucial ingredients. Without speaking for any of the participants, we suggest the following six principles for an empirically guided process to improve civil justice.

1. The impetus for new case management strategies should come from judges. In 1990, Congress passed the Civil Justice Reform Act, which sought to “simplify” federal civil case management in a number of ways. From the perspective of many judges, the reforms — which had been developed with little judicial input — were not well-suited to address the problems they perceived. Without either the substantive or political benefits of broad judicial buy-in, relatively few judges implemented the reforms in the way the architects had envisioned. The entire effort was widely viewed as unsuccessful. But in recent years, a second generation of reformers — mostly judges — has taken a different approach, developing reform ideas organically and starting with small-scale efforts that are (in most instances) subsequently evaluated by scholars. This approach has won a growing number of converts to the cause.

2. Case management reforms should be carefully evaluated. Most judges, in our experience, are interested in considering reforms, but they want to see clear and convincing evidence about the effects of reform. This means something more than just generating descriptive statistics; it requires good data and a well-crafted experimental design, capable of truly testing both the intended and unintended consequences of reform. Randomized controlled trials are ideal, but many types of “natural” experiments work, as well.¹ Helpfully, a cadre of scholars has emerged in recent years who are both interested and skilled in this type of collaborative investigation.

3. “Big data” is coming to court systems, and this can be a boon for the evaluation of case management innovations. Traditionally, scholars who wished to empirically analyze court processes had two choices: They could read and “code” the voluminous filings in a sample of cases, or they could rely on sketchy aggregate statistics compiled by court administrators. This was slow, labor-intensive work, and the limited number of cases studied limited the depth and generalizability of the analyses.

But in recent years, most courts have either adopted electronic filing systems or have taken significant steps in that direction. These data systems have the power to transform court research; so long as access is not curtailed by paywalls, it is now often feasible to find online the “universe” of relevant cases and draw out meaning from textual analysis. The two studies in our footnote each made use of hundreds of thousands of case dockets from publicly available online data.

It is often possible for scholars to use web scraping and other techniques to amass big datasets on case activity. But it is far better for judges and scholars to collaborate in the development of datasets, with user agreements that prevent data abuses while also improving the quality of the data scholars use.

4. Scholars, for their part, need judicial co-pilots. If the law is a maze, legal procedure is a labyrinth. The great danger of turning scholars loose on sophisticated, detailed datasets on court activities is the danger of simplistic analysis. It is obviously important for scholars to understand legal process and terminology; less obviously, it is vital for them to understand the hidden meanings behind the data. For example, a key indicator of a “successful” reform is often case duration — other things being equal, we generally assume that a faster case resolution is a better outcome. In the Los Angeles court system, duration is recorded as the time that elapses from the first case filing to the last entry in the docket. But there may be an entry, perhaps a year after a notice of settlement is filed by the parties, that simply represents a judge checking on whether all settlement sums have been paid so that dismissal can be entered. Without the aid of an interpreter-judge, the system (and the researcher) may incorrectly measure the true case length. Nearly all “measurement” variables used by researchers need this sort of judicial input to minimize misinterpretations and miscalibrations of the data.

5. Broad cooperation among those working on reforms can only be helpful. Although civil justice systems substantially vary across the country, and (due to local rules) often vary significantly even across counties, the commonalities are greater than the differences. There are pockets of reform in many states and federal districts; expanding these pockets and accelerating the rate of reform will be facilitated if the reformers are in contact with one another, sharing ideas, strategies, and empirical results and avoiding as much as possible the reinvention of the wheel.

6. It is valuable to have a big picture — with details. The case management reform vision of Judges Kuhl and Highberger fits wonderfully in this strategy. On one hand, they have managed to capture some universal principles of reform that are applicable to all civil courts. On the other, they have created a detailed schema to fill in the content of how the broad principles can be translated into specific innovations — and testable hypotheses. Experimentation within something like the Kuhl-Highberger schema provides an excellent agenda for collaborative innovation and evaluation going forward.

RICHARD SANDER is a Distinguished Professor of Law at UCLA and co-director of the UCLA-RAND Center on Law and Policy.

ERIC HELLAND is the Podlich Professor of Economics at Claremont McKenna College and an affiliated member of the RAND Graduate School faculty.

See Main Article: Toward Fairer, Quicker, Cheaper Litigation: A Unified Theory of Case Management by Carolyn B. Kuhl & William F. Highberger

Footnote:

1. For example, each of us recently conducted studies, using data from a large universe of Los Angeles County civil cases, that examined patterns across time and across judges and found two recent innovations to be highly effective. See Eric Helland & Minjae Yun, More Talk Less Conflict: Evidence from Requiring Informal Discovery Conferences (unpublished manuscript) (on file with authors); Richard Sander et al., Methods for Assessing Civil Justice Reform: The Case of “Meet and Confer” Requirements for California Demurrers, in Rethinking Case Management and the Process of Civil Justice Reform, RAND Institute for Civil Justice (2023), available at https://www.rand.org/pubs/conf_proceedings/CFA2386-1.html.