Recruitment, effort, and retention effects of performance contracts for civil servants: Experimental evidence from Rwandan primary schools

Last registered on October 23, 2018

Pre-Trial

General Information

Title

RCT ID

AEARCTR-0002565

Initial registration date

December 18, 2017

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published

December 19, 2017, 6:42 PM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated

October 23, 2018, 5:05 PM EDT

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Country

Rwanda

Region

Primary Investigator

Name

Andrew Zeitlin

Affiliation

Georgetown University

Contact Primary Investigator

Other Primary Investigator(s)

PI Name

Pieter Serneels

PI Affiliation

University of East Anglia

Contact Investigator

PI Name

Owen Ozier

PI Affiliation

World Bank

Contact Investigator

PI Name

Clare Leaver

PI Affiliation

University of Oxford

Contact Investigator

Additional Trial Information

Status

Completed

Start date

2015-10-01

End date

2018-03-31

Keywords

Education, Firms & Productivity, Labor

Additional Keywords

JEL code(s)

Secondary IDs

Abstract

The ability to recruit, elicit effort from, and retain civil servants is a central challenge of state capacity in developing countries. Nowhere is this more evident than in the education sector, where rising access to government schooling has failed to translate into hoped-for learning gains, even as teacher salaries account for the bulk of expenditure on education and a large part of the civil service payroll (Das et al, 2017). Many developing country governments obtain poor skill and effort levels in return for their expenditure on the teaching workforce: for example, the World Bank's Service Delivery Indicators for Uganda suggests that only 20 percent of primary school teachers have mastery of their content, while they are absent from school an average of 27 percent of the time (Wane and Martin 2013).

Despite growing evidence of the contractual determinants of effort among existing teachers, little is known about how to select the best staff. Hanushek and Rivkin (2006) highlight that teacher quality is only weakly predicted by formal qualifications and other ex-ante observable characteristics. This leaves open the question of whether incentive contracts might not only elicit effort on the job, but also attract more skilled and intrinsically motivated teachers. Lazear (2003) and Rothstein (2015) argue that performance contracts may be a cost effective means to attract high quality teachers, but to date there exists no experimental evidence on these labor-market effects of performance pay among teachers, or, for that matter, other civil servants in developing or developed countries.

To state our research questions, we sketch a simple framework that underpins our intervention. The production function for student learning depends on, inter alia, two teacher characteristics: skill (pre-determined) and effort (a strategic choice). Teacher utility is decreasing in effort and increasing in both money income and student learning. We refer to the relative importance of student learning in this utility function as a teacher’s intrinsic motivation.

This project seeks to evaluate not only the incentive effect (on effort) but also the selection effects (on skill and intrinsic motivation) of pay-for-performance (P4P) contracts. We have designed a two-tiered experiment to answer three primary research questions:
1. Can P4P improve teacher performance, and so contribute to student learning gains?
2. How effective are P4P contracts at recruiting effective (skilled and intrinsically motivated) teachers, particularly in rural areas?
3. Do P4P contracts help to retain effective teachers?
The hypothesized link from P4P contracts to the selection of teachers and their effort decisions on the job builds on a principal-agent model. In the simple ‘moral hazard’ variant of this model, P4P improves teacher effort. Richer models allow for selection effects on skill (e.g., Lazear 2000, 2003, Rothstein 2015): P4P is predicted to encourage more highly skilled individuals to join and remain in post. The existence of selection effects on intrinsic motivation is an open question.

These questions are answered through a two-tiered randomized, controlled trial, undertaken in the actual recruitment of civil-service teaching jobs listed for the 2016 school year. Further details are provided below.

External Link(s)

Registration Citation

Citation

Leaver, Clare et al. 2018. "Recruitment, effort, and retention effects of performance contracts for civil servants: Experimental evidence from Rwandan primary schools." AEA RCT Registry. October 23. https://doi.org/10.1257/rct.2565-5.0

Former Citation

Sponsors & Partners

Experimental Details

Interventions

Intervention(s)

We test the selection and incentive effects of P4P using a two-tiered RCT undertaken in the actual recruitment of civil-service teaching jobs in 2016. Both tiers of this experiment are built around the comparison of two contracts, managed by Innovations for Poverty Action in coordination with the Rwanda Education Board (REB), on top of teacher salaries. The first of these is a pay-for-performance contract, which pays RWF 100,000 (approximately 15 percent of annual salary) to the top 20 percent of upper-primary teachers as measured by a composite of teacher input metrics (presence, preparation, and pedagogy) and a student learning outcome metric based on Barlevy and Neal's (2012) pay-for-percentile metric. The second is a fixed wage contract that provides RWF 20,000 to all upper-primary teachers.

Intervention (Hidden)

Implementation of our intervention has proceeded as follows. Following a pilot in 2015, the two-tiered experiment started in the run-up to the 2016 academic year. The first tier randomized advertised jobs to contracts. New primary teaching posts in five core curricular subjects across six districts were allocated to P4P or fixed wage status (a total of 594 posts). The randomization took place at district-by-subject level. Since few teachers apply for jobs in multiple districts and prospective teachers are only eligible for posts if they have the relevant subject-family qualification, one can view district-subject family qualification pairs as distinct labor markets. As we discuss in the sample size section below, we have 18 distinct labor markets, and these are allocated to either a P4P, Fixed Wage, or Mixed treatment.

This first tier randomization was followed by an advertising campaign to increase awareness of the new posts and their associated contracts, including organization of ‘job fairs’ at Teacher Training Colleges. Extensive data on potential applicants were collected at these job fairs. Advertisements also took place over the radio, in person at District Education Offices, and through dissemination of printed materials in district capitals of the study districts.

The second tier randomized schools to contracts. A school was included in the sample if the new post was filled and assigned to upper primary grades (total 164 schools, 313 recruits). Following a full baseline survey in February 2016, sample schools were assigned to either P4P or fixed wage. All upper primary teachers within each school received the new contract. At individual applicant level, this amounted to re-randomization and hence a change to the initial assignment for some new recruits. All new recruits were paid a ‘retention bonus’ of 80,000 RWF to ensure that new contracts strictly dominated those advertised at the first tier. Teacher payments, determined in P4P schools by multiple teacher-input observations as well as beginning- and end-of-year student assessments, occurred in February 2017.

Supplementary funding has enabled us to continue the experiment in the same 164 schools for the 2017 school year. Together with measurement of teacher retention outcomes at the start of 2018, this will allow us to understand teacher retention impacts, since retention decisions for the 2016-17 school year transition took place before P4P winners were announced. A second year of exposure also creates more time for meaningful impacts on student learning gains and creates an opportunity to study dynamics in learning.

Intervention Start Date

2015-10-30

Intervention End Date

2018-02-28

Primary Outcomes

Primary Outcomes (end points)

Our two-tiered experimental design distinguishes between the actual contract experienced by hired teachers and the contract advertised to potential applicants. This design allows estimation of impacts of: (a) experienced P4P on the student learning gains achieved by 1,600 teachers in 164 schools—thereby answering Research Question 1; (b) advertised P4P on application volumes and applicant characteristics among 1,881 applicants to positions across the 6 districts, and on characteristics and performance of actual hires among the 313 successfully placed applicants—answering Research Question 2; and (c) experienced P4P on retention rates of both new and incumbent teachers—answering Research Question 3. Our approach enables us to isolate the selection effect in Research Question 2 by estimating the impact of advertised P4P, holding constant experienced P4P.

A blinded dataset will be used to finalize choices of primary outcomes and associated empirical specifications.

Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)

See above. A blinded dataset will be used to finalize choices of secondary outcomes and associated empirical specifications.

Secondary Outcomes (explanation)

Experimental Design

This study will test the selection and incentive effects of P4P using a two-tiered RCT undertaken in the actual recruitment of civil-service teaching jobs. This design draws on the 'surprise' two-stage randomizations of Karlan and Zinman (2008), Ashraf, Berry, and Shapiro (2010), and Cohen and Dupas (2010) in credit-market and public-health contexts. In the first tier, P4P versus fixed wage contracts were randomized via lottery at the labor-market level and advertised to potential applicants. In the second tier, contracts were (re)randomized at the school level (for schools that received new teachers in upper primary) to determine final offers to incumbent and newly recruited teachers. The re-randomization assigned contracts to either P4P or a fixed wage top-up, as in the first tier, but with new recruits also paid a ‘retention bonus’ of RWF 80,000 to ensure that new contracts strictly dominated those initially advertised in the first tier randomization.

Experimental Design Details

See hidden intervention section.

Randomization Method

Randomization was done in office by computer.

Randomization Unit

The first experimental tier is randomized at the labor-market level (district-by-subject); the second tier is randomized at the school level.

Was the treatment clustered?

Yes

Experiment Characteristics

Sample size: planned number of clusters

First tier. Treatment was initially assigned at the district-by-subject level (there are five subjects: math, science, English, Kinyarwanda, and social studies). These 30 district-by-subject pairs were assigned to either P4P or FW. As it turned out, districts actually solicited applications at the district-by-subject-family level (there are three subject families: math and science; modern languages; and social studies). It is these 18 district-by-subject-family pairs that we think of distinct labor markets, and which form our clusters.

Second tier. The 164 schools that received teachers through this process were randomized into experienced P4P or FW contracts. These schools represent our clusters in the second tier of the experiment.

Sample size: planned number of observations

The size of the applicant pool under advertised P4P and FW contracts is an object of study. Recruitment yielded a total of 1,881 applications and 313 upper-primary placements in study districts. Applicants placed in upper-primary grades were traced to 164 schools, which were enrolled and randomized into realized P4P or FW conditions. Sampled pupils for assessments were in excess of 14,000 per round (including pupils in grade-subject-streams taught by incumbent teachers as well as new recruits).

Sample size (or number of clusters) by treatment arms

First tier. Of the 18 labor markets in the first tier of the experiment, 7 were assigned to the P4P treatment (modern language teaching in Gatsibo and Kirehe, math and science teaching in Kayonza and Nygatore, and social studies teaching in Ngoma, Nygatore, and Rwamagama); 7 were assigned to the Fixed Wage treatment (modern language teaching in Kayonza and Rwamagama, math and science teaching in Kirehe and Ngoma, and social studies teaching in Gatsibo, Kayonza, and Kirehe); and 4 were assigned to a Mixed treatment (modern language teaching in Ngoma and Nygatore, and math and science teaching in Gatsibo and Rwamagama). To illustrate this Mixed treatment, an individual living in Ngoma with a qualification to teach modern languages could have applied to the modern languages pool, in which they would have been eligible for either advertised post in English on a Fixed Wage contract, or an advertised post in Kinyarwanda on a P4P contract. In contrast, someone with this qualification living in Gatsibo applying for a modern languages post might have been offered either an English or Kinyarwanda job, but both would have been on a P4P contract.

Second tier. Of the 164 schools in the second tier of the experiment, 85 were assigned to experienced P4P and 79 were assigned to experienced FW contracts.

Minimum detectable effect size for main outcomes (accounting for sample design and clustering)

Supporting Documents and Materials

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

IRB