Recruitment, effort, and retention effects of performance contracts for civil servants: Experimental evidence from Rwandan primary schools
Last registered on February 14, 2018

Pre-Trial

Trial Information
General Information
Title
Recruitment, effort, and retention effects of performance contracts for civil servants: Experimental evidence from Rwandan primary schools
RCT ID
AEARCTR-0002565
Initial registration date
December 18, 2017
Last updated
February 14, 2018 6:48 AM EST
Location(s)
Region
Primary Investigator
Affiliation
Georgetown University
Other Primary Investigator(s)
PI Affiliation
University of East Anglia
PI Affiliation
World Bank
PI Affiliation
University of Oxford
Additional Trial Information
Status
On going
Start date
2015-10-01
End date
2018-03-31
Secondary IDs
Abstract
The ability to recruit, elicit effort from, and retain civil servants is a central challenge of state capacity in developing countries. Nowhere is this more evident than in the education sector, where rising access to government schooling has failed to translate into hoped-for learning gains, even as teacher salaries account for the bulk of expenditure on education and a large part of the civil service payroll (Das et al, 2017). Many developing country governments obtain poor skill and effort levels in return for their expenditure on the teaching workforce: for example, the World Bank's Service Delivery Indicators for Uganda suggests that only 20 percent of primary school teachers have mastery of their content, while they are absent from school an average of 27 percent of the time (Wane and Martin 2013).

Despite growing evidence of the contractual determinants of effort among existing teachers, little is known about how to select the best staff. Hanushek and Rivkin (2006) highlight that teacher quality is only weakly predicted by formal qualifications and other ex-ante observable characteristics. This leaves open the question of whether incentive contracts might not only elicit effort on the job, but also attract more skilled and intrinsically motivated teachers. Lazear (2003) and Rothstein (2015) argue that performance contracts may be a cost effective means to attract high quality teachers, but to date there exists no experimental evidence on these labor-market effects of performance pay among teachers, or, for that matter, other civil servants in developing or developed countries.

To state our research questions, we sketch a simple framework that underpins our intervention. The production function for student learning depends on, inter alia, two teacher characteristics: skill (pre-determined) and effort (a strategic choice). Teacher utility is decreasing in effort and increasing in both money income and student learning. We refer to the relative importance of student learning in this utility function as a teacher’s intrinsic motivation.

This project seeks to evaluate not only the incentive effect (on effort) but also the selection effects (on skill and intrinsic motivation) of pay-for-performance (P4P) contracts. We have designed a two-tiered experiment to answer three primary research questions:
1. Can P4P improve teacher performance, and so contribute to student learning gains?
2. How effective are P4P contracts at recruiting effective (skilled and intrinsically motivated) teachers, particularly in rural areas?
3. Do P4P contracts help to retain effective teachers?
The hypothesized link from P4P contracts to the selection of teachers and their effort decisions on the job builds on a principal-agent model. In the simple ‘moral hazard’ variant of this model, P4P improves teacher effort. Richer models allow for selection effects on skill (e.g., Lazear 2000, 2003, Rothstein 2015): P4P is predicted to encourage more highly skilled individuals to join and remain in post. The existence of selection effects on intrinsic motivation is an open question.

These questions are answered through a two-tiered randomized, controlled trial, undertaken in the actual recruitment of civil-service teaching jobs listed for the 2016 school year. Further details are provided below.
External Link(s)
Registration Citation
Citation
Leaver, Clare et al. 2018. "Recruitment, effort, and retention effects of performance contracts for civil servants: Experimental evidence from Rwandan primary schools." AEA RCT Registry. February 14. https://www.socialscienceregistry.org/trials/2565/history/25798
Experimental Details
Interventions
Intervention(s)
We test the selection and incentive effects of P4P using a two-tiered RCT undertaken in the actual recruitment of civil-service teaching jobs in 2016. Both tiers of this experiment are built around the comparison of two contracts, managed by Innovations for Poverty Action in coordination with the Rwanda Education Board (REB), on top of teacher salaries. The first of these is a pay-for-performance contract, which pays RWF 100,000 (approximately 15 percent of annual salary) to the top 20 percent of upper-primary teachers as measured by a composite of teacher input metrics (presence, preparation, and pedagogy) and a student learning outcome metric based on Barlevy and Neal's (2012) pay-for-percentile metric. The second is a fixed wage contract that provides RWF 20,000 to all upper-primary teachers.
Intervention Start Date
2015-10-30
Intervention End Date
2018-02-28
Primary Outcomes
Primary Outcomes (end points)
Our two-tiered experimental design distinguishes between the actual contract experienced by hired teachers and the contract advertised to potential applicants. This design allows estimation of impacts of: (a) experienced P4P on the student learning gains achieved by 1,600 teachers in 164 schools—thereby answering Research Question 1; (b) advertised P4P on application volumes and applicant characteristics among 1,881 applicants to positions across the 6 districts, and on characteristics and performance of actual hires among the 313 successfully placed applicants—answering Research Question 2; and (c) experienced P4P on retention rates of both new and incumbent teachers—answering Research Question 3. Our approach enables us to isolate the selection effect in Research Question 2 by estimating the impact of advertised P4P, holding constant experienced P4P.

A blinded dataset will be used to finalize choices of primary outcomes and associated empirical specifications.
Primary Outcomes (explanation)
Secondary Outcomes
Secondary Outcomes (end points)
See above. A blinded dataset will be used to finalize choices of secondary outcomes and associated empirical specifications.
Secondary Outcomes (explanation)
Experimental Design
Experimental Design
This study will test the selection and incentive effects of P4P using a two-tiered RCT undertaken in the actual recruitment of civil-service teaching jobs. This design draws on the 'surprise' two-stage randomizations of Karlan and Zinman (2008), Ashraf, Berry, and Shapiro (2010), and Cohen and Dupas (2010) in credit-market and public-health contexts. In the first tier, P4P versus fixed wage contracts were randomized via lottery at the labor-market level and advertised to potential applicants. In the second tier, contracts were (re)randomized at the school level (for schools that received new teachers in upper primary) to determine final offers to incumbent and newly recruited teachers. The re-randomization assigned contracts to either P4P or a fixed wage top-up, as in the first tier, but with new recruits also paid a ‘retention bonus’ of RWF 80,000 to ensure that new contracts strictly dominated those initially advertised in the first tier randomization.
Experimental Design Details
See hidden intervention section.
Randomization Method
Randomization was done in office by computer.
Randomization Unit
The first experimental tier is randomized at the labor-market level (district-by-subject); the second tier is randomized at the school level.
Was the treatment clustered?
Yes
Experiment Characteristics
Sample size: planned number of clusters
First tier. Treatment was initially assigned at the district-by-subject level (there are five subjects: math, science, English, Kinyarwanda, and social studies). These 30 district-by-subject pairs were assigned to either P4P or FW. As it turned out, districts actually solicited applications at the district-by-subject-family level (there are three subject families: math and science; modern languages; and social studies). It is these 18 district-by-subject-family pairs that we think of distinct labor markets, and which form our clusters.

Second tier. The 164 schools that received teachers through this process were randomized into experienced P4P or FW contracts. These schools represent our clusters in the second tier of the experiment.
Sample size: planned number of observations
The size of the applicant pool under advertised P4P and FW contracts is an object of study. Recruitment yielded a total of 1,881 applications and 313 upper-primary placements in study districts. Applicants placed in upper-primary grades were traced to 164 schools, which were enrolled and randomized into realized P4P or FW conditions. Sampled pupils for assessments were in excess of 14,000 per round (including pupils in grade-subject-streams taught by incumbent teachers as well as new recruits).
Sample size (or number of clusters) by treatment arms
First tier. Of the 18 labor markets in the first tier of the experiment, 7 were assigned to the P4P treatment (modern language teaching in Gatsibo and Kirehe, math and science teaching in Kayonza and Nygatore, and social studies teaching in Ngoma, Nygatore, and Rwamagama); 7 were assigned to the Fixed Wage treatment (modern language teaching in Kayonza and Rwamagama, math and science teaching in Kirehe and Ngoma, and social studies teaching in Gatsibo, Kayonza, and Kirehe); and 4 were assigned to a Mixed treatment (modern language teaching in Ngoma and Nygatore, and math and science teaching in Gatsibo and Rwamagama). To illustrate this Mixed treatment, an individual living in Ngoma with a qualification to teach modern languages could have applied to the modern languages pool, in which they would have been eligible for either advertised post in English on a Fixed Wage contract, or an advertised post in Kinyarwanda on a P4P contract. In contrast, someone with this qualification living in Gatsibo applying for a modern languages post might have been offered either an English or Kinyarwanda job, but both would have been on a P4P contract.

Second tier. Of the 164 schools in the second tier of the experiment, 85 were assigned to experienced P4P and 79 were assigned to experienced FW contracts.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
IRB
INSTITUTIONAL REVIEW BOARDS (IRBs)
IRB Name
Innovations for Poverty Action
IRB Approval Date
2014-12-29
IRB Approval Number
1502
IRB Name
Rwanda Ministry of Education Research Review Committee
IRB Approval Date
2015-06-02
IRB Approval Number
MINEDUC/S&T/308/2015
Post-Trial
Post Trial Information
Study Withdrawal
Intervention
Is the intervention completed?
No
Is data collection complete?
Data Publication
Data Publication
Is public data available?
No
Program Files
Program Files
Reports and Papers
Preliminary Reports
Relevant Papers