Intervention (Hidden)
We recruited about one-third of trial court judges in Pakistan (N=800) to participate in a judge training course where they will receive access to a judge support chatbot and get training in using this tool. The study will employ a randomized controlled trial design, where about half of the judges (N=400) will be randomized into the first round of the course in February-March 2024, and the other half (N=400) will be randomized into the second round of the course in October-November 2024. The technology underlying the judge chatbot is ChatGPT, and the tool is comparable to free access to an OpenAI ChatGPT+ account. The training is a course on the most useful functionalities of ChatGPT for legal work. The sample includes judges from across Pakistan, ensuring diverse representation.
There are five data sources. First, we have data from surveys of the judges. There are four survey waves with the following approximate timeline. The baseline is in December 2023 before both intervention waves, and we have a second in March 2024 after the first intervention wave, a third in September 2024 before the second intervention wave, and a fourth in December 2024 after the second intervention wave. judges in both treatment groups take all surveys. The baseline survey includes detailed information on judge characteristics. The subsequent surveys include follow-up questions on the use of AI tools, attitudes toward AI and tech, perceived work performance, and work-life balance. The follow-up surveys will be short to avoid attrition.
Second, we have data from the course activities and assignments. This includes records on course attendance and participation, recorded in the Zoom logs and lecture software, and assignments recorded via assignment submission software. The assignments will be graded for quality by teaching assistants.
Third, we will have the full record of the judge’s interactions with the chatbot. That will include interactions during the lectures, as well as outside the lectures on homework assignments. Most importantly, it includes subsequent interactions, even after the course is done, to see if the judges use the chatbot in their work or outside of work. We can examine the text inputs and text outputs. We can see if the judges uploaded documents and their interactions, including summarization or information retrieval.
Fourth, we have administrative data provided by the courts. That will allow us to track changes in case backlog, case disposal rates, and the shares of various case outcomes. For example, we can look at the share of cases dismissed, number of hearings, case duration, win rates for plaintiff/defendant, and appeal rate.
Fifth, we have judicial opinions and writings provided by the judges. We can see if measurable features of the rulings change, in terms of the language features, style of legal reasoning, or how cases are cited. We can match the text of the rulings to the specific judge’s text outputs from the chatbot, to understand when and how judges used those outputs. We will construct
The primary instruments for data collection include the customized web interface for the judge support chatbot, course implementation software, and standardized survey forms for capturing quantitative and qualitative changes in judicial processes. Additionally, surveys and interviews may be conducted to gather subjective experiences and feedback from participating judges. All instruments and detailed methodologies will be elaborated in the relevant appendices, ensuring comprehensive documentation of the study's approach and tools.