Evidence on Performance-Based Pay from School-Based Nutrition Programs in Rural China

Last registered on March 01, 2017

Pre-Trial

Trial Information

General Information

Title
Evidence on Performance-Based Pay from School-Based Nutrition Programs in Rural China
RCT ID
AEARCTR-0001469
Initial registration date
March 01, 2017

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
March 01, 2017, 12:01 PM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Region

Primary Investigator

Affiliation
Stanford Medical School

Other Primary Investigator(s)

PI Affiliation
PI Affiliation
Assistant Professor, School of Economics, Renmin University of China
PI Affiliation
PI Affiliation

Additional Trial Information

Status
Completed
Start date
2011-08-01
End date
2012-05-31
Secondary IDs
Abstract
A large literature examines performance pay for managers in the private sector, but little is known about performance pay for managers in public sector bureaucracies. In this paper, researchers study performance incentives rewarding school administrators for reducing anemia among their students. This study is part of a larger randomized control trial “Paying for Performance and Cost Effectiveness of Strategies to Combat Anemia in China.” which was implemented in Northwest China. The trial involved a total of 300 primary schools across Qinghai, Gansu and Shaanxi provinces. It was designed as a cluster randomized trial with 300 schools randomly allocated to one of five performance incentive arms: Arm 1: Budget Support and Information Only (65 Schools); Arm 2: Large Anemia Reduction Incentive (65 Schools); Arm 3: Small Anemia Reduction Incentive (40 Schools); Arm 4: Test Score Incentive (65 schools); and Arm 5: Dual Incentive Principals (65 schools). Each of these arms had two orthogonally assigned block-grant arms: a “small” block grant group and a “large” block grant group, resulting in 10 experimental groups.

The study discussed in this paper focuses only Arms 1, 2, and 3. Randomly assigning 170 schools to three performance incentive levels and two orthogonal sizes of unconditional grants, researchers analyze performance pay and its complementarity with discretionary resources. They find that both large incentives and larger unconditional grants reduced anemia substantially, but incentives were more cost-effective. Performance incentives led administrators to innovate by working with parents, mitigating potentially offsetting compensatory behavior among households. Researchers also find that larger unconditional grants completely crowded-out the effect of incentives. Their findings suggest that performance incentives can be effective in bureaucratic environments – but also that discretionary resources can fully crowd-out their effect.

The remaining 130 of 300 schools were allocated to two other experimental Arms, where school administrators were offered incentives to based on test scores and “dual” incentives based on both anemia reduction and test score improvement. The results for these study arms are reported in a companion paper (Miller, G., Sylvia, S., Vera-Hernandez, M. (2015). “Multiple Tasks and Multiple Rewards: Experimental Evidence on Performance Incentives, Alignment and Complementarity from Chinese Schools.” Working paper).

Note, the original trial was registered on October 12, 2012 in the J-Pal Hypothesis Registry because the AEA RCT Registry did not exist at that time (https://www.povertyactionlab.org/Hypothesis-Registry).
External Link(s)

Registration Citation

Citation
Luo, Renfu et al. 2017. "Evidence on Performance-Based Pay from School-Based Nutrition Programs in Rural China." AEA RCT Registry. March 01. https://doi.org/10.1257/rct.1469-1.0
Former Citation
Luo, Renfu et al. 2017. "Evidence on Performance-Based Pay from School-Based Nutrition Programs in Rural China." AEA RCT Registry. March 01. https://www.socialscienceregistry.org/trials/1469/history/14590
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
This study is part of a larger randomized control trial “Paying for Performance and Cost Effectiveness of Strategies to Combat Anemia in China,” which was implemented in Northwest China and was designed to test outcome-based performance pay based on school-level anemia prevalence given to school principals as a means of improving the effectiveness of school-based anemia reduction programs, including how performance pay based on anemia prevalence compares to more traditional rewards based on student exam scores, how the two types of rewards affect student achievement, and the complementarity of the two types of rewards in their effects on anemia rates and academic performance, as well as how results varied by the size of incentives provided. The trial involved a total of 300 primary schools across Qinghai, Gansu and Shaanxi provinces. The experiment was designed as a cluster randomized trial with 300 schools randomly allocated to one of five treatment arms: Arm 1: Budget Support and Information Only (65 Schools); Arm 2: Large Anemia Reduction Incentive (65 Schools); Arm 3: Small Anemia Reduction Incentive (40 Schools); Arm 4: Test Score Incentive (65 schools); and Arm 5: Dual Incentive Principals (65 schools). Within each of these groups, half were assigned to receive a small subsidy and half a large subsidy.

This part of the study includes 170 schools assigned to Arms 1, 2, and 3. Once schools were allocated to one of these treatment arms, the principal of each school receiving a small incentive, a large incentive, or no incentive for reducing anemia prevalence in their schools.

Comparison group: In 65 schools, principals did not receive any incentive to reduce anemia; they received only budget support and information. (Arm 1 of the original trial)

Large performance incentives: In 65 schools, principals received a larger incentive of 125 yuan (US$20) for each student who was anemic at the start of the intervention and no longer anemic at the end. Under this incentive structure, it was feasible for principals in this group to earn incentives equivalent to two additional months of salary. (Arm 2 of the original trial)

Small performance incentives: In 40 schools, principals received 12.5 Chinese yuan (US$2) for each student who was anemic at the start of the intervention and no longer anemic at the end. (Arm 3 of the original trial)

Across each of the three incentive groups, all schools were randomly assigned to receive small or large black grants in order to examine the effect of different amounts of resources on the prevalence of anemia. Although these funds were given in the context of the nutrition program, principals were free to reallocate them to other school functions, and receiving these funds did not depend on reducing anemia prevalence.
Intervention Start Date
2011-10-01
Intervention End Date
2012-05-31

Primary Outcomes

Primary Outcomes (end points)
Primary outcomes: hemoglobin concentration and a dichotomous indicator for anemia status
Secondary outcomes: Index: Supplements and Food, with Sub-Indeces Supplements, Food, Food at School, and Food at Home; Index: Information, with Sub-Indeces Information to Students and Informational to Households
Primary Outcomes (explanation)
Secondary outcomes were used to assess behavioral responses underlying changes in primary outcomes (i.e., actions taken by administrators (i.e., provision of supplements or food and communication with households) and subsequent responses among students and their parents (i.e., student consumption of iron-rich foods, direct iron supplementation, communication between parents and schools about anemia and its nutritional basis). See analysis plan for details.

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
This study is part of a larger randomized controlled trial “Paying for Performance and Cost Effectiveness of Strategies to Combat Anemia in China.” In the original trial, researchers began with 36 counties officially designated by the Chinese government as “poverty counties” in five regions (prefectures) in western China (Haidong in Qinghai Province, Dingxi, Tianshui, and Longnan in Gansu Province, and Ankang in Shaanxi Province). In August 2011, researchers conducted a canvass survey in each county to construct a list of all rural primary schools and the number of students enrolled in each. Restricting the sampling frame to primary schools with 150-300 students total, they randomly selected 300 of 1,410 eligible schools for inclusion in the study (and limited selection to one school per township). The sample size was based on power calculations conducted using data from primary schools in the same region of China. Within study schools, researchers randomly sampled 50 fourth and fifth grade students from each school (10-to-11-year-old students). Researchers chose these grades to select students sufficiently old to provide meaningful survey responses – but also sufficiently young to be generally prepubescent. Researchers conducted physical exams and collected data from students from other grades at baseline to obfuscate the focus on fourth and fifth graders.

The original experiment was designed as a cluster-randomized trial using a 5x2 crosscutting design. After conducting the baseline survey, researchers provided all school administrators with information about anemia, and schools were randomly assigned to one of 10 experimental cells. The five incentive groups arms were: Arm 1: Budget Support and Information Only (65 Schools); Arm 2: Large Anemia Reduction Incentive (65 Schools); Arm 3: Small Anemia Reduction Incentive (40 Schools); Arm 4: Test Score Incentive (65 schools); and Arm 5: Dual Incentive Principals (65 schools). The planned methodology and specifications for analyzing the impacts of the program are outlined in the Pre-Analysis Plan. Each of these arms had two orthogonally assigned block-grant arms: a “small” block grant group and a “large” block grant group, resulting in 10 experimental groups.

This part of the study includes only Arm 1, referred to as Group A: a group without incentives; Arm 2, referred to as Group C: a “large” incentive group; and Arm 3, referred to as Group B: a “small” incentive group. As noted, each of these arms has two orthogonally assigned block-grant arms: a “small” block grant group (Group 1: 85 schools; Group A1: 32 schools; Group B1: 20 schools; Group C1: 33 schools) and a “large” block grant group (Group 2: 85 schools; Group A2: 33 schools; Group B2: 20 schools; Group C2: 32 schools). The reference group is the default policy (education about anemia coupled with a modest resource transfer and no incentives, Group A1). To improve power, researches used stratified randomization. Using joint quintiles of the baseline distribution of school-level hemoglobin concentration and combined standardized math and Chinese exam scores – yielding 25 strata, researchers randomized cell assignment within each stratum. This analysis takes this randomization procedure into account, conditioning on stratum fixed effects.

In the large incentive group - Group C - researchers offered school administrators financial incentives to be paid as private income according to the net reduction in number of students identified as anemic between the beginning and end of the school year, with 125 yuan (RMB) ($19.40) per student reduction in anemia (i.e., two months of a school administrator’s annual salary for a 50-percent reduction in anemia). Researchers did not reveal the identity of anemic students. The small incentive group - Group B - was identical to the Group C except that the incentive was 12.5 RMB ($1.95) per student reduction in anemia (i.e., 0.2 additional months of annual salary for 50-percent reduction in anemia). When school administrators signed incentive contracts, they were told the (implied) number of anemic students in their schools (student identity was not revealed). Contracts were written using official letterhead of the Chinese Academy of Sciences (a government agency) and counter-signed by the deputy director of the implementing research center (school administrators signed two copies of the contract, one of which they kept). All interventions were implemented in partnership with local education bureaus, signaling that the project was sanctioned by local governments.

The small block grant group - Group 1 - was 0.3 RMB ($0.05) per student per day for school administrators to purchase vitamins for each student to take daily. The large block grant - Group 2 - was 0.7 RMB ($0.11) per student per day. In total, Group 1 schools received 7,452 yuan ($1,164) on average and Group 2 schools received 17,388 yuan ($2,717). These grants were given to schools in two installments (at the beginning of the program and half way through the school year). Although funds were given in the context of nutrition, administrators were free to allocate these to other school functions at their discretion.

Prior to revealing treatment assignment, researchers provided health education about nutrition and anemia to all school administrators in the study. The baseline survey was conducted in September 2011 and follow-up survey in May 2012 (i.e., at the beginning and end of the 2011-2012 academic year), collecting detailed information on students, households, school administrators, and schools. As part of Student Surveys, researchers interviewed all sampled students at their schools, collecting information on student background, health behaviors related to anemia, school activities, and general health. Students were given standard food frequency questionnaires to collect information about food consumption at school and at home over the past week. Researchers measured student blood hemoglobin (Hb) concentration at the time of the student survey. Nurses from the Medical School of Xi’an Jiaotong University accompanied study enumerators, collecting finger-prick blood samples to analyze on-site (at schools) using HemoCue Hb 201+ assessment systems. As part of Household Surveys, for each sampled student, researchers collected information about interactions between parents and the school, household income and assets, health-related expenditures, expenditures on food and information on other household members, focusing on household characteristics that students would be unlikely to know themselves. As part of School Administrator Surveys, researchers interviewed school administrators (bureaucrats) at three different points in time: before and after school administrators were told about the incentive contract and block grant to which they were assigned and again at endline. At baseline, school administrators provided information about their background, job history, salary, and compensation as well as perceptions of professional responsibilities and anemia knowledge. Researchers also measured the intrinsic and pro-social motivation of administrators. Following their participation in the training session on anemia (conducted 3 weeks after the baseline survey) administrators were given a second short survey to measure their understanding of the training material. Through School Surveys, researchers collected basic information from schools (enrollment, staffing, facilities, finances, and meal provision) and teachers (teacher characteristics, communication with parents, and teaching practices).

For primary outcomes (hemoglobin concentration and a dichotomous indicator for anemia status), researchers test eight null hypotheses: five for treatment main effects and their interactions and three additional ones – that the small and large incentives have the same average effect, that the large incentive and the large block grant have the same average effect, and that the average effect of the large incentive in presence of a large grant is zero. Therefore researchers adjust p-values to control the Family Wise Error Rate (FWER) using the free step-down resampling method of Westfall and Young (1993), which accounts for the dependency of the data. For secondary outcomes, researchers adjust p-values according to the total number of tests within a family of outcomes (the number of outcomes in the family times five – the number of treatment coefficients in each regression). Secondary outcomes are behavioral responses underlying changes in primary outcomes (actions taken by administrators (provision of supplements or food and communication with households) and subsequent responses among students and their parents (student consumption of iron-rich foods, direct iron supplementation, communication between parents and schools about anemia and its nutritional basis).

Finally, researchers examine the cost-effectiveness of each intervention combination, considering both the sub-sample of children anemic at baseline and the full sample of children, and present both “programmatic” cost-effectiveness (direct monetary program costs to the implementing organization) and social cost-effectiveness calculations. Researchers calculate total social costs as the sum of programmatic costs, the cost of public funds, and costs incurred by households in response to the interventions.

As noted earlier, the full experiment involved 300 schools. The remaining 130 schools were allocated to two other experimental Arms, where school administrators were offered incentives to based on test scores and “dual” incentives based on both anemia reduction and test score improvement. The results for these study arms are reported in a companion paper (Miller, G., Sylvia, S., Vera-Hernandez, M. (2015). “Multiple Tasks and Multiple Rewards: Experimental Evidence on Performance Incentives, Alignment and Complementarity from Chinese Schools.” Working paper).
Experimental Design Details
Randomization Method
Computer
Randomization Unit
School (for treatment) and student (for testing/survey)
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
Original randomized control trial: 300 schools. This study: 170 of these 300 schools
Sample size: planned number of observations
NA; randomization was clustered
Sample size (or number of clusters) by treatment arms
Arm 1/Group A (No Administrator Incentive): 65 schools with 32 schools in Group A1: Small Block Grant and 33 Schools in Group A2: Large Block Grant
Arm 3/Group B (Small Anemia Reduction Incentive): 40 schools, with 20 schools in Group B1: Small Block Grant and 20 schools in Group B2: Large Block Grant
Arm 2/Group C (Large Anemia Reduction Incentive): 65 schools, with 33 schools in Group C1: Small Block Grant and 32 schools in Group C2: Large Block Grant

Arm 4 (Test Score Incentive) - part of the original trial, but not included in this study - 65 schools, with 32 schools with small block grants and 33 schools with large block grants

Arm 5 (Dual Incentive) - part of the original trial, but not included in this study - 65 schools, with 33 schools with small block grants and 32 schools with large block grants

Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
IRB

Institutional Review Boards (IRBs)

IRB Name
Stanford University
IRB Approval Date
2012-09-25
IRB Approval Number
No. 19748
Analysis Plan

Analysis Plan Documents

P4P Anemia Pre-Analysis Plan

MD5: de6742081aa71ef8f353912cafdf2419

SHA1: 60fd733d66610ef78e520125038e0e6a1f144dee

Uploaded At: September 03, 2016

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
Yes
Intervention Completion Date
May 31, 2012, 12:00 +00:00
Data Collection Complete
Yes
Data Collection Completion Date
May 31, 2012, 12:00 +00:00
Final Sample Size: Number of Clusters (Unit of Randomization)
Original randomized control trial: 300 schools. This study: 170 of the 300 schools
Was attrition correlated with treatment status?
No
Final Sample Size: Total Number of Observations
This study includes 21,107 students
Final Sample Size (or Number of Clusters) by Treatment Arms
Arm 1/Group A (No Administrator Incentive): 65 schools with 32 schools in Group A1: Small Block Grant and 33 Schools in Group A2: Large Block Grant Arm 3/Group B (Small Anemia Reduction Incentive): 40 schools, with 20 schools in Group B1: Small Block Grant and 20 schools in Group B2: Large Block Grant Arm 2/Group C (Large Anemia Reduction Incentive): 65 schools, with 33 schools in Group C1: Small Block Grant and 32 schools in Group C2: Large Block Grant Arm 4 (Test Score Incentive) - part of the original trial, but not included in this study - 65 schools, with 32 schools with small block grants and 33 schools with large block grants Arm 5 (Dual Incentive) - part of the original trial, but not included in this study - 65 schools, with 33 schools with small block grants and 32 schools with large block grants
Data Publication

Data Publication

Is public data available?
Yes

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Abstract
A large literature examines performance pay for managers in the private sector, but little is known about performance pay for managers in public sector bureaucracies. In this paper, researchers study performance incentives rewarding school administrators for reducing anemia among their students. Randomly assigning 170 schools to three performance incentive levels and two orthogonal sizes of unconditional grants, researchers analyze performance pay and its complementarity with discretionary resources. They find that both large incentives and larger unconditional grants reduced anemia substantially, but incentives were more cost-effective. Performance incentives led administrators to innovate by working with parents, mitigating potentially offsetting compensatory behavior among households. Researchers also find that larger unconditional grants completely crowded-out the effect of incentives. Their findings suggest that performance incentives can be effective in bureaucratic environments – but also that discretionary resources can fully crowd-out their effect.
Citation
Luo, Renfu, Grant Miller, Scott Rozelle, Sean Sylvia, and Marcos Vera-Hernndez. "Can Bureaucrats Really be Paid like CEOs? School Administrator Incentives for Anemia Reduction in Rural China." NBER Working Paper No. 21302, June 2015.

Reports & Other Materials