Subjective versus Objective Performance Pay for Teachers

Last registered on March 23, 2019

Pre-Trial

Trial Information

General Information

Title
Subjective versus Objective Performance Pay for Teachers
RCT ID
AEARCTR-0003835
Initial registration date
March 21, 2019

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
March 23, 2019, 8:19 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Region

Primary Investigator

Affiliation
University of Chicago

Other Primary Investigator(s)

PI Affiliation
Lahore School of Economics

Additional Trial Information

Status
On going
Start date
2017-10-02
End date
2019-12-31
Secondary IDs
Abstract
This study looks at the effects of providing teachers raises based on their students' test scores versus their principal's rating of them as compared to a flat raise. We measure the effect on teacher behavior in and out of the classroom, teacher retention, and student performance. While subjective performance pay is easier to implement and can be more successful at addressing multi-tasking concerns, there is a worry that principals may be biased against certain groups or show favoritism toward teachers they are socially connected to. We look for evidence of gender discrimination and favoritism toward connected teachers as well as heterogeneity in treatment effects by more or less biased principals. Finally, we introduce an additional randomization as to how often principals observe certain teachers to vary the principals' accuracy of information. We look to see if principals accurately update and if teachers in response work harder when their effort is more observable.
External Link(s)

Registration Citation

Citation
Andrabi, Tahir and Christina Brown. 2019. "Subjective versus Objective Performance Pay for Teachers." AEA RCT Registry. March 23. https://doi.org/10.1257/rct.3835-1.0
Former Citation
Andrabi, Tahir and Christina Brown. 2019. "Subjective versus Objective Performance Pay for Teachers." AEA RCT Registry. March 23. https://www.socialscienceregistry.org/trials/3835/history/43982
Experimental Details

Interventions

Intervention(s)
A. Incentive Treatments
Control: Flat raise - All teachers will receive the same raise (about 7%) at the end of the calendar year. This is equal to about twice the cost of living inflation.

Treatment 1a: Subjective Performance Raise - All teachers will be rated by their direct supervisor (usually a principal or vice principal) on the teacher's "effort in improving students' academic performance'' on a scale from 0-100. Supervisors will give a short justification for their rating. Within the school, teachers will then be ranked according to this subjective score, placed into 5 bins and given a raise from 0-15% based on their ranking.

Treatment 1b: Subject Performance Raise - All teachers will be rated by their direct supervisor (usually a principal or vice principal) on a number of criteria determined by HR, such as timeliness, communication and professional integrity on a scale from 0-100. Supervisors will give a short justification for their rating. Within the school, teachers will then be ranked according to this subjective score, placed into 5 bins and given a raise from 0-15% based on their ranking.

Treatment 2: Objective Performance Raise - All teachers receive a raise based on percentile value-added (Barlevy and Neal, 2016) -- their students' average percentile within their lagged percentile comparison group on the standardized exam at the end of each semester. Teachers will then be ranked within the school based on this measure of value added, placed into 5 bins and receive a raise varying from 0-15%.

On average the raise percentage is the same across all treatments.

B. Observation Treatment

Control: Status quo – Teachers observed at the normal frequency by their supervisor, typically once a semester

Treatment: Frequent observation – Principals are requested to observe teachers for at least 10 minutes once every three weeks
Intervention Start Date
2018-02-01
Intervention End Date
2019-04-30

Primary Outcomes

Primary Outcomes (end points)
The primary outcomes of interest are student test performance, measured in January 2019, and teacher effort, measured by attendance and clock in and out time from March 2018-November 2018 and scores on the CLASS rubric from videos of classes.
Primary Outcomes (explanation)
Student test performance will be calculated using within class-subject z-scores, averaging across all subjects. We will also calculate IRT scores.

Secondary Outcomes

Secondary Outcomes (end points)
Secondary outcomes are student satisfaction and socio-emotional skills from a survey conducted in January 2019, test performance broken down by subject (English, Math, Urdu and Science) student retention, teacher time use and satisfaction from a survey conducted in March 2019.
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
We randomly assign teachers to receive a variety of incentive contracts and cross randomize with how often their manager observes them. The key specification will look at the effect of each scheme on test scores and teacher effort. We will also look at the interaction with being observed more to determine whether that increases the perceived "bite" of the subjective incentives and therefore, teacher effort in that treatment.

We will take the data to a classical principal-agent model to understand how the perceived noisiness and distortion of each incentive system helps us understand the choices teachers make in terms of the extent and direction of effort change.

In March we will conduct a survey to understand what teachers believed about each incentive system. If teachers interpreted both treatments similarly, we will collapse those two treatments.

Note: The listed design details are being pre-specified after the treatment implementation is complete but before endline data was finished being collected. PIs had reviewed midline student grade outcomes and video data prior to pre-specification.
Experimental Design Details
Randomization Method
Randomization was conducted live via Stata during a meeting with the implementing partner and research staff.
Randomization Unit
The unit of randomization for the incentives treatment was first at the school level (Treatment 1b vs everything else) and then the randomization was at the school-grade section level. A grade section is, for example, the primary, middle or high school section within a school. The unit of randomization for the observation treatment was first as the school level, for which principals would be trained on the observation rubric and then randomization was conducted at the teacher level for which teachers would receive more frequent observations.
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
There are 155 schools and 300 grade-sections.
Sample size: planned number of observations
There are 5,000 teachers and 55,000 students
Sample size (or number of clusters) by treatment arms
Incentive Treatment --
First level of randomization: (randomized at school level): 88 schools assigned to treatment 1b, 67 schools assigned to next level of randomization
Second level of randomization: (randomized at grade section level): 43 assigned to treatment 1a, 43 assigned to treatment 2, 43 assigned to control

Observation Treatment --
First level of randomization (at the school level): 99 assigned to be part of the observation treatment and 56 part of the control
Second level of randomization (at the teacher level): 2100 assigned to receive treatment and 2400 assigned to control
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Pairwise randomization by baseline test performance was used for the incentive randomization, which generally performs better than traditional stratification for smaller samples (Bruhn and Mckenzie, 2009). We will control for individual baseline outcomes (student or teacher depending on the type of outcome) and school baseline outcomes, randomization strata fixed effects and exam version for student test outcomes. Given the previous year to year correlation of our outcomes and intra-cluster correlation, we expect to be powered to detect an effect of 0.1 sd in for student outcomes and 0.15 sd effect for teacher outcomes for the incentive treatments. For the monitoring treatment, because we have randomized at the teacher level, we can detect smaller effects of 0.08 sd on student outcomes and 0.1 sd for teacher outcomes.
IRB

Institutional Review Boards (IRBs)

IRB Name
Pomona College Institutional Review Board
IRB Approval Date
2018-03-28
IRB Approval Number
#03282018TA-TB

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials