Effect of Court Auditors on Judicial Decisions

Last registered on July 19, 2023

Pre-Trial

Trial Information

General Information

Title
Effect of Court Auditors on Judicial Decisions
RCT ID
AEARCTR-0011751
Initial registration date
July 09, 2023

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
July 19, 2023, 12:08 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Region

Primary Investigator

Affiliation
HEC Montreal

Other Primary Investigator(s)

PI Affiliation
New York Law School
PI Affiliation
Northwestern-Kellogg
PI Affiliation
University of Pennsylvania

Additional Trial Information

Status
In development
Start date
2023-07-10
End date
2024-07-10
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
Every day in immigration courts across the US, judges hold hearings to determine whether the foreign nationals in their dockets (technically called respondents) should be removed from the country or, instead, allowed to stay. In these hearings, judges have latitude in choosing procedural steps that may, eventually, result in the foreign nationals being removed. The goal of this research is to assess whether judges respond to being observed by law students and other trained observers and, in particular, whether being observed makes a judge more motivated to provide flexibility to the individuals appearing before them and more responsive to the requests for continuances or other needs.

To this end, we will train Observers (law students and others) to attend these public hearings and take notes about the procedures used. Observation may be in person or remote through the WEBEX viewer.

The procedure will be to randomize Observers to hearings, and then compare the procedural choices made in observed hearings (treatment) with those made in hearings not observed (controls). In addition, we want to know whether any difference in procedural choice might be due only to the judge’s heightened concern about making “improper” choices while being audited, or whether the judge might also respond to the judges’ supervisor knowledge about the particular judge rendering “improper” decisions while being audited. The EOIR requires the judges and their support staff to code each procedural choice in the individual hearings. This data is released pursuant to standing FOIA requests on the EOIR website. We will be focusing on the observations in three New York City area immigration courts which is one of the largest adjudication workloads in the national system.

The number of treated hearings will be determined based on the available budget. We expect an Observer-day to cost about $75, and an Observer audits one hearing a day. EOIR procedures divide regular removal hearings into status dockets called "Master Calendars" and merits hearings called "Individual Hearings." Master Calendars vary in size but average between 30 and 50 individuals in a four hour sitting. Our observations are only of Master Calendars. In these hearings (or dockets), the judge may take a wide variety of actions primarily related to scheduling or setting deadlines for applications such as asylum relief. Cases can be concluded during a Master Calendar hearing.

Given the available budget ($20,000), we expect to be able to “treat” about 266 hearings. The number of control hearings will exceed the number of treated hearings.
External Link(s)

Registration Citation

Citation
Benson, Lenni et al. 2023. "Effect of Court Auditors on Judicial Decisions ." AEA RCT Registry. July 19. https://doi.org/10.1257/rct.11751-1.0
Experimental Details

Interventions

Intervention(s)
The experimental subjects are approximately 70 judges in the New York City immigration courts. The goal of the research is to assess whether these judges respond to being observed by law students or other trained observers and, in particular, whether being observed makes a judge responsive to requests for more time for the individual to find legal representation (not provided at government expense in this forum) or to prepare applications for relief. To this end, we will train Observers (law students and others) to attend these public hearings and take notes about the procedures used. Observation may be in person or remote through the WEBEX viewer.
The procedure will be to randomize Observers to hearings, and then compare the procedural choices made in observed hearings (treatment) with those made in hearings not observed (controls). In addition, we want to know whether any difference in procedural choice might be due only to the judge’s heightened concern about making “improper” choices while being audited, or whether the judge might also respond to the judges’ supervisor knowledge about the particular judge rendering “improper” decisions while being audited.
There are three groups:
1. Judges who are audited in a particular hearing
2. Judges who are audited in a particular hearing and told that the auditor’s presence will be reported to the judge’s supervisor
3. Control group: judges who are not audited on that particular hearing

Procedural choices on all immigration hearings (outcomes on both treated and controls) are made publicly available by the DOJ through their EOIR website. The EOIR requires the judges and their support staff to code each procedural choice in the individual hearings. This data is released pursuant to standing FOIA requests on the EOIR website. We will be focusing on the observations in three New York City area immigration courts which is one of the largest adjudication workloads in the national system.
Intervention Start Date
2023-07-10
Intervention End Date
2023-10-10

Primary Outcomes

Primary Outcomes (end points)
We will collect data both at the hearing level and regarding the future evolution of audited cases, as they unfold. These outcomes will be recorded from the EOIR administrative data.


Outcomes at hearing level
Whether the audited hearing is
1. the last hearing
2. adjourned
3. if adjourned, different adjournment reasons

Outcomes at the case level
1. Case duration
2. Number of hearings
3. Appeal rate
4. Decision type (e.g., administrative closure, remove, etc.)
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
1. Length of the hearing
Secondary Outcomes (explanation)

Experimental Design

Experimental Design

The experimental subjects are approximately 70 judges in the New York City immigration courts. The goal of the research is to assess whether these judges respond to being observed by law students or other trained observers and, in particular, whether being observed makes a judge more motivated to provide flexibility to the individuals appearing before them and more responsive to the requests for continuances or other needs. To this end, we will train Observers (law students and others) to attend these public hearings and take notes about the procedures used. Observation may be in person or remote through the WEBEX viewer.
The procedure will be to randomize Observers to hearings, and then compare the procedural choices made in observed hearings (treatment) with those made in hearings not observed (controls). In addition, we want to know whether any difference in procedural choice might be due only to the judge’s heightened concern about making “improper” choices while being audited, or whether the judge might also respond to the judges’ supervisor knowledge about the particular judge rendering “improper” decisions while being audited.
There are three groups:
1. Judges who are audited in a particular hearing
2. Judges who are audited in a particular hearing and told that the auditor’s presence will be reported to the judge’s supervisor
3. Control group: judges who are not audited on that particular hearing

Procedural choices on all immigration hearings (outcomes on both treated and controls) are made publicly available by the DOJ through their EOIR website. The EOIR requires the judges and their support staff to code each procedural choice in the individual hearings. This data is released pursuant to standing FOIA requests on the EOIR website.
Experimental Design Details
Randomization Method
We will randomly assign auditors to judges using a web interface that, for a given day in which the auditor declares his/her availability, randomizes the auditor to a hearing among the judges who hold a hearing on that day.
Randomization Unit
The randomization unit is at the judge-hearing level.
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
The number of treated hearings will be determined based on the available budget. We expect an Observer-day to cost about $75, and an Observer audits one hearing a day. Given the available budget ($20,000), we expect to be able to “treat” about 266 hearings. EOIR procedures divide regular removal hearings into status dockets called "Master Calendars"" and merits hearings called "Individual Hearings." Master Calendars vary in size but average between 30 and 50 individuals in a four hour sitting. Our observations are only of Master Calendars.
In these sittings or dockets, the judge may take a wide variety of actions primarily related to scheduling or setting deadlines for applications such as asylum relief. Cases can be concluded during a Master Calendar hearing. The number of control hearings will exceed the number of treated hearings.

We will cluster at the judge-hearing level, both for treated and for control hearings.
Sample size: planned number of observations
Each hearing usually involves 30-50 cases
Sample size (or number of clusters) by treatment arms
Target number of treated hearings: about 266.
Target number of control hearings: as many as available from the EOIR database during the sampling period.

1. 133 hearings of judges who are audited
2. 133 hearings of judges who are audited and told that the auditor’s presence will be reported to the judge’s supervisor
3. As many as possible judged in the control group: judges who are not audited on that particular hearing
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
IRB

Institutional Review Boards (IRBs)

IRB Name
New York Law School Institutional Review Board
IRB Approval Date
2023-06-07
IRB Approval Number
N/A

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials