Overcoming Adverse Selection: Understanding Sorting in Response to Performance Incentives

A. Incentive Treatments
Control: Flat raise - All teachers will receive the same raise (about 7%) at the end of the calendar year. This is equal to about twice the cost of living inflation.
Treatment 1a: Subjective Performance Raise - All teachers will be rated by their direct supervisor (usually a principal or vice principal) on the teacher's "effort in improving students' academic performance'' on a scale from 0-100. Supervisors will give a short justification for their rating. Within the school, teachers will then be ranked according to this subjective score, placed into 5 bins and given a raise from 0-15% based on their ranking.
Treatment 1b: Subject Performance Raise - All teachers will be rated by their direct supervisor (usually a principal or vice principal) on a number of criteria determined by HR, such as timeliness, communication and professional integrity on a scale from 0-100. Supervisors will give a short justification for their rating. Within the school, teachers will then be ranked according to this subjective score, placed into 5 bins and given a raise from 0-15% based on their ranking.
Treatment 2: Objective Performance Raise - All teachers receive a raise based on percentile value-added (Barlevy and Neal, 2016) -- their students' average percentile within their lagged percentile comparison group on the standardized exam at the end of each semester. Teachers will then be ranked within the school based on this measure of value added, placed into 5 bins and receive a raise varying from 0-15%.
On average the raise percentage is the same across all treatments.

B. Observation Treatment
Control: Status quo – Teachers observed at the normal frequency by their supervisor, typically once a semester
Treatment: Frequent observation – Principals are requested to observe teachers for at least 10 minutes once every three weeks

C. Information Treatment
Mid-way into the intervention, a fraction of teachers were told their mid-term performance under different incentive schemes. This was randomized at the teacher level.

All teachers were provided information about the treatment assignment of all other schools in the system in the event they wanted to change schools.

2018-02-01

2019-10-01

There are several key outcomes of interest: 1. Teachers contract choice: elicited at baseline and endline in an incentive-compatible choice exercise 2. Teacher job choice: location of the teacher in December 2018 when the contracts were paid out

n order to understand teacher quality (before, during, and after the intervention) we will use student test performance, measured in June 2018, January 2018, June 2018, January 2019, and June 2019, and teacher effort, measured by attendance and clock in and out time from March 2018-November 2018 and scores on the CLASS rubric from videos of classes. We will also use student satisfaction and socio-emotional skills from a survey conducted in January 2019.

We randomly assign teachers to receive a variety of incentive contracts and cross randomize with how often their manager observes them and whether they learn about their performance half way through the intervention. The key specification will look at whether higher quality teachers (as measured by baseline value-added and teacher effort) i). choose performance pay in incentivized choice experiments and/or ii). sort into performance pay schools. We will see if this varies by the riskiness associated with each contract and the information teachers have about their own performance. We will also disentangle the extent of sorting that takes place based on the teachers' unincentivized quality "type" versus sorting on their treatment effect "type"(ie. how much they respond to incentives).

We will then take these results to a two-sided choice model to understand the extent of adverse selection that can be corrected by introducing these contracts given the information principals already have about their own teachers.

Note: The design details presented here were outlined in Ms. Brown's orals' prospectus in March 2018 prior to any midline or endline data being collected but after the beginning of the treatment intervention. Prospectus available upon request. Further details about a companion study were previously registered here: https://www.socialscienceregistry.org/trials/3835

Randomization was conducted live via Stata during a meeting with the implementing partner and research staff.

The unit of randomization for the incentives treatment was first at the school level (Treatment 1b vs everything else) and then the randomization was at the school-grade section level. A grade section is, for example, the primary, middle or high school section within a school. The unit of randomization for the observation treatment was first as the school level, for which principals would be trained on the observation rubric and then randomization was conducted at the teacher level for which teachers would receive more frequent observations.

Yes

There are 155 schools and 300 grade-sections.

There are 5,000 teachers and 55,000 students.

Incentive Treatment --
First level of randomization: (randomized at school level): 88 schools assigned to treatment 1b, 67 schools assigned to next level of randomization
Second level of randomization: (randomized at grade section level): 43 assigned to treatment 1a, 43 assigned to treatment 2, 43 assigned to control

Observation Treatment --
First level of randomization (at the school level): 99 assigned to be part of the observation treatment and 56 part of the control
Second level of randomization (at the teacher level): 2100 assigned to receive treatment and 2400 assigned to control

Pairwise randomization by baseline test performance was used for the incentive randomization, which generally performs better than traditional stratification for smaller samples (Bruhn and Mckenzie, 2009). We will control for individual baseline outcomes (student or teacher depending on the type of outcome) and school baseline outcomes, randomization strata fixed effects and exam version for student test outcomes. Given the previous year to year correlation of our outcomes and intra-cluster correlation, we expect to be powered to detect an effect of 0.1 sd in for student outcomes and 0.15 sd effect for teacher outcomes for the incentive treatments. For the monitoring treatment, because we have randomized at the teacher level, we can detect smaller effects of 0.08 sd on student outcomes and 0.1 sd for teacher outcomes.