Improving Legal Training: the Impact of Social-Emotional Learning and Class Monitoring on Judicial Performance

First, we are implementing an online monitoring program in which classes are observed by a member of the agency’s pedagogical team and teachers receive personalized feedback after each monitored session. Each class in the monitoring group of this intervention is monitored by a member of Judicial Academy's (AMAG's) pedagogical team. The monitor joins the day-long classes for a total of one hour divided into four slots of fifteen minutes spread across the duration of the class. After each class, the teacher meets with the monitor to receive and discuss her feedback during a 30-minute follow-up meeting. Classes in the control group are not monitored and teachers do not receive any feedback.

Second, we are applying online self-reflection exercises to the AMAG's students. Once a month, students will be asked to reflect and write about topics related to the work of judicial professionals. The treatment group will be asked to reflect about their personal experiences whereas the control treatment group will be asked about the same topic but reflecting on others' experience as a judge or prosecutor. These questions will change every month from a list of 9 topics, each of which has a treatment and a control question. To ensure that the writing task is effective, a minimum of words will be required and answers will be peer-reviewed. At the end of the month, students will be asked to rate their peers’ (anonymized) responses to the exercise in terms of how useful they are as advice for future judges and prosecutors. Each student will rate two pairs of responses, selecting their preferred response for each pair.

2020-05-23

2020-12-31

There are 3 main sets of outcomes of interest:
1. Educational attainment: this includes student’s grades, attendance, participation in class, and satisfaction with the course.
2. Empathy and bias: at the end of the program, an exit survey will evaluate a set of outcomes of judges and prosecutors. These outcomes will include a dictator game, a redistribution game, an honesty test, an Implicit Association Test and the decision to choose a book on a topic related to morality, law and economics, gender in law or judicial independence.
3. Professional performance: these outcomes refer to judges’ and prosecutors’ performance on the job. We aim to measure their productivity via the following indicators: case clearance rate, case length, fraction of decisions appealed or reversed, length and direction of judicial decisions, textual measures of implicit bias, and future promotion.

Immediate short-term outcomes will also include program participation and the grade of the peer-review ratings on their colleague’s responses to the social-emotional learning interventions. Secondary outcomes might also include the income and employment of those litigants and firms who might be affected by the decisions of these judges and prosecutors.

The research design consists in a randomized controlled trial (RCT) of two main interventions: online monitoring of teachers during classes, and social-emotional learning (SEL) exercises written by the students.

Online monitoring will take place every class (twice a month), and will be randomized at the class level. Those students whose teachers are monitored in the first round will remain in the monitoring group in future courses. The teacher will likely change every month (course), yet the new teacher will also be monitored if the previous teacher was monitored too.

The SEL exercises will take place once a month, when students will be asked to reflect and write about topics related to the work of judicial professionals. At the end of the month, students will review two pairs of anonymized responses from their peers and select their preferred ones from each pair. The SEL topics will not only be randomized across students in each course; the order in which students complete such topics will also be randomized. This will allow us to study the comparative effect of each of these topics, and thus understand which interventions are most helpful to improve educational attainment and professional performance of judges and prosecutors.

Randomization done by a computer.

We need to distinguish the two interventions. For the monitoring intervention, the unit of randomization will be the class. For the SEL intervention, the randomization unit is the student.

Yes

For the monitoring intervention, there are 22 classes (clusters) over the period of 9 courses. The classes remain on the same group (treatment or control) during the entire program. We cluster at the class level, program level, course type, and location. Given that some locations only have 1 class per program, level and course, we aggregate all of the single locations in one common group.

For the SEL intervention, there are 600 students who receive 10 topics to write about in random order, one topic for each course from course two onwards. For each topic, whether the student is assigned the treatment question or the control question is also randomly assigned. The treatment/control randomization for each question is stratified based on the following variables: level of the program, location of student, class, and profession (judge or prosecutor).

SEL: 600 students for the SEL intervention over the period of 8 months. Monitoring: There are 22 classes for the monitoring of teachers over the period of 9 months, affecting 600 students in total.

Monitoring: 11 classes in treatment, 11 classes in control; over the period of 9 months.
SEL: 60 students per treatment arm stratified by level of the program, location, class, and profession (judges or prosecutors) over the period of 8 months.