| Field | Before | After |
|---|---|---|
| Field Abstract | Before We integrate a specialized AI course into the Pakistan Judicial Academy's "Technology and Law” initiative. We will design and build an AI-based judge support tool that empowers judges to search, cite, and summarize the history of Pakistan precedents as well as newly submitted briefs and other documents. The system, especially with the associated training, is protected against plagiarism, hallucination, and providing false citations. The tool and training will be provided in the context of a randomized field experiment, equipping about one-third of Pakistan's trial court judges (approximately 800 judges) with generative AI technology, and associated training and support. Our research will evaluate the effect of AI technology and training on judge performance, including self-reported work satisfaction, case disposal rates, and quality measures constructed from written rulings. The findings of our study are expected to shed light on the potential of generative AI to bolster state capabilities and judicial productivity worldwide. *Update to Trial - Rerandomization in April 2024 After the initial random assignment, the introduction of the password-protected JudgeGPT, which was designed specifically to prevent spillovers in the experiment, sparked significant interest among judges who had not previously registered for the course but wished to access JudgeGPT, which was not available to them. An additional 205 judges expressed interest in participating in the course to gain access to JudgeGPT. To manage this surge of interest—which could enhance our statistical power—while preserving the study's integrity, we decided against simply adding these new applicants to our control group, namely to our control Batch 2, as the newly interested judges had not been randomly selected. Therefore, we concluded that a second randomization was necessary to uphold the experiment's integrity and to increase the study's statistical power, allowing for the inclusion of 1,185 judges instead of the originally registered 980. The second stage of randomization, therefore, took place on April 26, 2024. We first merged the original Batch 2 (492 judges) with Batch 3 of new registrations (n = 205). These new 697 judges were divided into New Batch 2 (n =349), who will take the course in September 2024, and Final Batch 3 (n = 348), who will take the course in November 2024 and will serve as our control group. A detailed update document, which includes flowcharts and explanatory notes to elucidate the re-randomization process, is provided in the Analysis Plan section. | After We integrate a specialized AI course into the Pakistan Judicial Academy's "Technology and Law” initiative. We will design and build an AI-based judge support tool that empowers judges to search, cite, and summarize the history of Pakistan precedents as well as newly submitted briefs and other documents. The system, especially with the associated training, is protected against plagiarism, hallucination, and providing false citations. The tool and training will be provided in the context of a randomized field experiment, equipping about one-third of Pakistan's trial court judges with generative AI technology and associated training and support. Our research will evaluate the effect of AI technology and training on judge performance, including AI usage, perceived AI usefulness, and quality measures constructed from written rulings. Our study's findings have the possibility to shed light on the potential of generative AI to bolster state capabilities and judicial productivity worldwide. *Update to Trial - Rerandomization in October 2024 The random assignment of judges was conducted in two distinct waves for registration of judges into JudgeGPT subscriptions. In February 2024, the first wave of registration saw 979 judges from Pakistan's lower courts sign up to participate in our experiment. We randomly assigned these 979 judges into two groups: 487 judges were allocated to the treatment group (Batch 1) and provided with access to the JudgeGPT subscription and GPT instruction course, while the remaining 492 judges were designated as the control group (Batch 2), scheduled to receive the same access in September 2024. This setup allows for a randomized control trial comparing the outcomes of Batch 1 and Batch 2. Following the initial random assignment, the introduction of the password-protected JudgeGPT, designed specifically to prevent spillovers, sparked considerable interest among judges who had not initially registered for the course but were nonetheless eager to participate but could not access GPT or the course. An additional 580 judges expressed interest in the course and the JudgeGPT tool. To preserve the study's integrity, we decided against adding these new applicants to our control group (Batch 2), as they were not randomly assigned. Therefore, a second randomization was conducted to maintain the integrity of the study and increase its statistical power, accommodating a total of 1559 judges instead of the initially registered 979. This means more than 50% of the trial court judges (court of first instance) in Pakistan registered to participate in our experiment. In October, the second wave of randomization, therefore, took place on October 23, 2024, for 580 judges. The 580 judges were randomly assigned to Batch 3 (n = 218), which will take the course in December 2024 and January 2025, and Batch 4 (n = 362). Batch 3 judges would get the same treatment as Batch 1 and 2: JudgeGPT course and JudgeGPT subscription. Batch 4, however, is further randomized into two subgroups: Batch 4a and Batch 4b. Batch 4a is randomly assigned to receive JudgeGPT training and a placebo course on Technology and Law in December 2024 and January 2025, along with a GPT subscription (and an anti-hallucination warning in GPT). Batch 4b will also take the generic Technology and Law course during the same period but will not receive a GPT subscription. The key difference is that Batch 4a will have access to the GPT subscription with a hallucination warning, while Batch 4b will not. Both groups, however, will attend the Generic Law and Technology classes at the same time that Batch 3 is receiving the JudgeGPT course. This will allow us to assess the impact of access to GPT tools on judges' learning and decision-making. Please see Figure 1 and other details in the Pre-analysis plan document for a summary of the experimental design and more details. |
| Field Last Published | Before May 26, 2024 03:46 PM | After November 01, 2024 06:19 AM |
| Field | Before | After |
|---|---|---|
| Field Document | Before |
After
Re-Randomization AI for Judges 24 October 2024.pdf
MD5:
4c13b047c84de2a801dc031ba70c8e43
SHA1:
8a0fa00de39995f8433da23f51972e881fbdec20
|
| Field Title | Before | After October 2024 Update Document with re-randomization of additional judges |
| Field | Value |
|---|---|
| Field Document |
Value
Update_Document_Rerandomization.pdf
MD5:
fa9919235a3babd58837cdc799e7d618
SHA1:
2b8f743562de04306da1893fc70c2b44f97c9a59
|
| Field Title | Value Update Document with details on Re-randomization |