Abstract
We integrate a specialized AI course into the Pakistan Judicial Academy's "Technology and Law” initiative. We will design and build an AI-based judge support tool that empowers judges to search, cite, and summarize the history of Pakistan precedents as well as newly submitted briefs and other documents. The system, especially with the associated training, is protected against plagiarism, hallucination, and providing false citations. The tool and training will be provided in the context of a randomized field experiment, equipping about one-third of Pakistan's trial court judges (approximately 800 judges) with generative AI technology, and associated training and support. Our research will evaluate the effect of AI technology and training on judge performance, including self-reported work satisfaction, case disposal rates, and quality measures constructed from written rulings. The findings of our study are expected to shed light on the potential of generative AI to bolster state capabilities and judicial productivity worldwide.
*Update to Trial - Rerandomization in April 2024
After the initial random assignment, the introduction of the password-protected JudgeGPT, which was designed specifically to prevent spillovers in the experiment, sparked significant interest among judges who had not previously registered for the course but wished to access JudgeGPT, which was not available to them. An additional 205 judges expressed interest in participating in the course to gain access to JudgeGPT. To manage this surge of interest—which could enhance our statistical power—while preserving the study's integrity, we decided against simply adding these new applicants to our control group, namely to our control Batch 2, as the newly interested judges had not been randomly selected. Therefore, we concluded that a second randomization was necessary to uphold the experiment's integrity and to increase the study's statistical power, allowing for the inclusion of 1,185 judges instead of the originally registered 980.
The second stage of randomization, therefore, took place on April 26, 2024. We first merged the original Batch 2 (492 judges) with Batch 3 of new registrations (n = 205). These new 697 judges were divided into New Batch 2 (n =349), who will take the course in September 2024, and Final Batch 3 (n = 348), who will take the course in November 2024 and will serve as our control group. A detailed update document, which includes flowcharts and explanatory notes to elucidate the re-randomization process, is provided in the Analysis Plan section.