Experimental Design
This study is part of a larger randomized controlled trial “Paying for Performance and Cost Effectiveness of Strategies to Combat Anemia in China.” In the original trial, researchers began with 36 counties officially designated by the Chinese government as “poverty counties” in five regions (prefectures) in western China (Haidong in Qinghai Province, Dingxi, Tianshui, and Longnan in Gansu Province, and Ankang in Shaanxi Province). In August 2011, researchers conducted a canvass survey in each county to construct a list of all rural primary schools and the number of students enrolled in each. Restricting the sampling frame to primary schools with 150-300 students total, they randomly selected 300 of 1,410 eligible schools for inclusion in the study (and limited selection to one school per township). The sample size was based on power calculations conducted using data from primary schools in the same region of China. Within study schools, researchers randomly sampled 50 fourth and fifth grade students from each school (10-to-11-year-old students). Researchers chose these grades to select students sufficiently old to provide meaningful survey responses – but also sufficiently young to be generally prepubescent. Researchers conducted physical exams and collected data from students from other grades at baseline to obfuscate the focus on fourth and fifth graders.
The original experiment was designed as a cluster-randomized trial using a 5x2 crosscutting design. After conducting the baseline survey, researchers provided all school administrators with information about anemia, and schools were randomly assigned to one of 10 experimental cells. The five incentive groups arms were: Arm 1: Budget Support and Information Only (65 Schools); Arm 2: Large Anemia Reduction Incentive (65 Schools); Arm 3: Small Anemia Reduction Incentive (40 Schools); Arm 4: Test Score Incentive (65 schools); and Arm 5: Dual Incentive Principals (65 schools). The planned methodology and specifications for analyzing the impacts of the program are outlined in the Pre-Analysis Plan. Each of these arms had two orthogonally assigned block-grant arms: a “small” block grant group and a “large” block grant group, resulting in 10 experimental groups.
This part of the study includes only Arm 1, referred to as Group A: a group without incentives; Arm 2, referred to as Group C: a “large” incentive group; and Arm 3, referred to as Group B: a “small” incentive group. As noted, each of these arms has two orthogonally assigned block-grant arms: a “small” block grant group (Group 1: 85 schools; Group A1: 32 schools; Group B1: 20 schools; Group C1: 33 schools) and a “large” block grant group (Group 2: 85 schools; Group A2: 33 schools; Group B2: 20 schools; Group C2: 32 schools). The reference group is the default policy (education about anemia coupled with a modest resource transfer and no incentives, Group A1). To improve power, researches used stratified randomization. Using joint quintiles of the baseline distribution of school-level hemoglobin concentration and combined standardized math and Chinese exam scores – yielding 25 strata, researchers randomized cell assignment within each stratum. This analysis takes this randomization procedure into account, conditioning on stratum fixed effects.
In the large incentive group - Group C - researchers offered school administrators financial incentives to be paid as private income according to the net reduction in number of students identified as anemic between the beginning and end of the school year, with 125 yuan (RMB) ($19.40) per student reduction in anemia (i.e., two months of a school administrator’s annual salary for a 50-percent reduction in anemia). Researchers did not reveal the identity of anemic students. The small incentive group - Group B - was identical to the Group C except that the incentive was 12.5 RMB ($1.95) per student reduction in anemia (i.e., 0.2 additional months of annual salary for 50-percent reduction in anemia). When school administrators signed incentive contracts, they were told the (implied) number of anemic students in their schools (student identity was not revealed). Contracts were written using official letterhead of the Chinese Academy of Sciences (a government agency) and counter-signed by the deputy director of the implementing research center (school administrators signed two copies of the contract, one of which they kept). All interventions were implemented in partnership with local education bureaus, signaling that the project was sanctioned by local governments.
The small block grant group - Group 1 - was 0.3 RMB ($0.05) per student per day for school administrators to purchase vitamins for each student to take daily. The large block grant - Group 2 - was 0.7 RMB ($0.11) per student per day. In total, Group 1 schools received 7,452 yuan ($1,164) on average and Group 2 schools received 17,388 yuan ($2,717). These grants were given to schools in two installments (at the beginning of the program and half way through the school year). Although funds were given in the context of nutrition, administrators were free to allocate these to other school functions at their discretion.
Prior to revealing treatment assignment, researchers provided health education about nutrition and anemia to all school administrators in the study. The baseline survey was conducted in September 2011 and follow-up survey in May 2012 (i.e., at the beginning and end of the 2011-2012 academic year), collecting detailed information on students, households, school administrators, and schools. As part of Student Surveys, researchers interviewed all sampled students at their schools, collecting information on student background, health behaviors related to anemia, school activities, and general health. Students were given standard food frequency questionnaires to collect information about food consumption at school and at home over the past week. Researchers measured student blood hemoglobin (Hb) concentration at the time of the student survey. Nurses from the Medical School of Xi’an Jiaotong University accompanied study enumerators, collecting finger-prick blood samples to analyze on-site (at schools) using HemoCue Hb 201+ assessment systems. As part of Household Surveys, for each sampled student, researchers collected information about interactions between parents and the school, household income and assets, health-related expenditures, expenditures on food and information on other household members, focusing on household characteristics that students would be unlikely to know themselves. As part of School Administrator Surveys, researchers interviewed school administrators (bureaucrats) at three different points in time: before and after school administrators were told about the incentive contract and block grant to which they were assigned and again at endline. At baseline, school administrators provided information about their background, job history, salary, and compensation as well as perceptions of professional responsibilities and anemia knowledge. Researchers also measured the intrinsic and pro-social motivation of administrators. Following their participation in the training session on anemia (conducted 3 weeks after the baseline survey) administrators were given a second short survey to measure their understanding of the training material. Through School Surveys, researchers collected basic information from schools (enrollment, staffing, facilities, finances, and meal provision) and teachers (teacher characteristics, communication with parents, and teaching practices).
For primary outcomes (hemoglobin concentration and a dichotomous indicator for anemia status), researchers test eight null hypotheses: five for treatment main effects and their interactions and three additional ones – that the small and large incentives have the same average effect, that the large incentive and the large block grant have the same average effect, and that the average effect of the large incentive in presence of a large grant is zero. Therefore researchers adjust p-values to control the Family Wise Error Rate (FWER) using the free step-down resampling method of Westfall and Young (1993), which accounts for the dependency of the data. For secondary outcomes, researchers adjust p-values according to the total number of tests within a family of outcomes (the number of outcomes in the family times five – the number of treatment coefficients in each regression). Secondary outcomes are behavioral responses underlying changes in primary outcomes (actions taken by administrators (provision of supplements or food and communication with households) and subsequent responses among students and their parents (student consumption of iron-rich foods, direct iron supplementation, communication between parents and schools about anemia and its nutritional basis).
Finally, researchers examine the cost-effectiveness of each intervention combination, considering both the sub-sample of children anemic at baseline and the full sample of children, and present both “programmatic” cost-effectiveness (direct monetary program costs to the implementing organization) and social cost-effectiveness calculations. Researchers calculate total social costs as the sum of programmatic costs, the cost of public funds, and costs incurred by households in response to the interventions.
As noted earlier, the full experiment involved 300 schools. The remaining 130 schools were allocated to two other experimental Arms, where school administrators were offered incentives to based on test scores and “dual” incentives based on both anemia reduction and test score improvement. The results for these study arms are reported in a companion paper (Miller, G., Sylvia, S., Vera-Hernandez, M. (2015). “Multiple Tasks and Multiple Rewards: Experimental Evidence on Performance Incentives, Alignment and Complementarity from Chinese Schools.” Working paper).