Courts of Tomorrow

Last registered on January 31, 2024

View Trial History

Pre-Trial

Trial Information

General Information

Title

Courts of Tomorrow

RCT ID

AEARCTR-0012906

Initial registration date

January 28, 2024

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published

January 31, 2024, 12:11 PM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Primary Investigator

Name

Sultan Mehmood

Affiliation

New Economic School Moscow

Contact Primary Investigator

Other Primary Investigator(s)

PI Name

Elliot Ash

PI Affiliation

ETH Zurich

Contact Investigator

Additional Trial Information

Status

In development

Start date

2024-01-01

End date

2025-12-31

Keywords

Governance, Other

Additional Keywords

Generative AI, Judiciary, Technology and Law

JEL code(s)

O33 Technological Change: Choices and Consequences • Diffusion Processes M53 Training K41 Litigation Process C45 Neural Networks and Related Topics

Secondary IDs

Prior work

This trial does not extend or rely on any prior RCTs.

Abstract

We integrate a specialized AI course into the Pakistan Judicial Academy's "Technology and Law” initiative. We will design and build an AI-based judge support tool that empowers judges to search, cite, and summarize the history of Pakistan precedents as well as newly submitted briefs and other documents. The system, especially with the associated training, is protected against plagiarism, hallucination, and providing false citations. The tool and training will be provided in the context of a randomized field experiment, equipping about one-third of Pakistan's trial court judges (approximately 800 judges) with generative AI technology, and associated training and support. Our research will evaluate the effect of AI technology and training on judge performance, including self-reported work satisfaction, case disposal rates, and quality measures constructed from written rulings. The findings of our study are expected to shed light on the potential of generative AI to bolster state capabilities and judicial productivity worldwide.

External Link(s)

Registration Citation

Citation

Ash, Elliot and Sultan Mehmood. 2024. "Courts of Tomorrow." AEA RCT Registry. January 31. https://doi.org/10.1257/rct.12906-1.0

Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Experimental Details

Interventions

Intervention(s)

We recruited about one-third of trial court judges in Pakistan to participate in a judge training course where they will receive access to a judge support chatbot and get training in using this tool. The study will employ a randomized controlled trial design, where about half of the judges will be randomized into a first round of the course, and the other half will be randomized into a second round about 8 months later.

Intervention Start Date

2024-02-19

Intervention End Date

2024-12-31

Primary Outcomes

Primary Outcomes (end points)

1. Course attendance and participation.
2. Use of the AI tool.
3. Judgment Quality using Judgments Texts Data. The Federal Judicial Academy will share judgment texts on each judge that will allow us to construct the following main text-based measures:.
4. Perceptions of AI Utility
5. Productivity. Judges’ expectations of AI's potential to improve their productivity and their support for using generative AI at work.

Primary Outcomes (explanation)

Outcomes 1 and 2 come from our intervention. They are used to estimate a “first stage” for takeup of the intervention.
Outcome 3 is from the Federal Judicial Academy (administrative data) which obtains a random sample of 50 judgements each before and after its training programs.
Outcomes 4-5 are survey perceptions provided by the judges. They allow us to estimate the subjective impact of AI technology and training.
The subsequent surveys include additional follow-up questions on the use of AI tools, attitudes toward AI and tech, perceived work performance, and work-life balance. The follow-up surveys will be short to avoid attrition.
We have data from the course activities and assignments. This includes records on course attendance and participation, recorded in the Zoom logs and lecture software, and assignments recorded via assignment submission software. The assignments will be graded for quality by teaching assistants.
We have the full record of the judge’s interactions with the chatbot. That will include interactions during the lectures, as well as outside the lectures on homework assignments. Most importantly, it includes subsequent interactions, even after the course is done, to see if the judges use the chatbot in their work or outside of work. We can examine the text inputs and text outputs. We can see if the judges uploaded documents and their interactions, including summarization or information retrieval.

Secondary Outcomes

Secondary Outcomes (end points)

1. Case Backlog and Disposal Rate. This administrative data will be procured from the Judicial Academy and the Pakistan Law and Justice Commission of Pakistan if and when it becomes available.
2. Workload Management. Measuring the number of different cases a judge works on at any given time, along with the hours spent on work.

Secondary Outcomes (explanation)

We hope to procure administrative data . That will allow us to track changes in case backlog, case disposal rates, and the shares of various case outcomes. For example, we can look at the share of cases disposed, conviction rates, court congestions, case duration, win rates for plaintiff/defendant, and appeal rate.

Experimental Design

The study will employ a randomized controlled trial design, where about half of the judges will be randomized into a first round of the course, and the other half will be randomized into a second round in the next course term.

Experimental Design Details

Not available

Randomization Method

Randomization is done by a computer.

Randomization Unit

We recruited about one-third of trial court judges in Pakistan (N=800) to participate. The study will employ a randomized controlled trial design, where about half of the judges (N=400) will be randomized into the first round of the course in February-March 2024, and the other half (N=400) will be randomized into the second round of the course in September--October 2024. Stratification is based on the time of enrollment in the course, the age of judges, and the district court in which the judges serve.

Was the treatment clustered?

Yes