Experimental Design
The evaluation employed a prospective experimental design, where program-eligible localities were randomly assigned to three statuses: receive the gender-uniform subsidy subprogram (treatment 1), receive the gender-differentiated subsidy subprogram (treatment 2), receive neither subprogram (control).
The effect of the program on outcomes of interest involve comparing treatment localities to control localities before and after the onset of the treatment (double differences). For some outcomes, only the single cross-sectional difference were examined. Given random assignment to treatment status, the single differencing will yield valid estimates of the program impact. The effects of the two individual treatments (the gender-uniform subsidy and the gender-differentiated subsidy) was determined by applying the same empirical strategy.
The selection of localities and school operators and their random assignment to the program followed a stepwise process. First, applicants submitted applications to the program with one or more localities where they want to set up schools. Second, the applicants and proposed localities were vetted separately based on their respective qualification criteria. Third, all qualifying applicants were contacted with the full list of qualifying localities and asked to select which localities they would like to set up schools; qualifying applicants can choose more than locality from this list, and unpicked qualifying localities were dropped. This “relinked list” of qualifying localities was provided to the impact evaluation team. The team then randomly assigned each locality from the list to one of three statuses: treatment 1, treatment 2, and control.
SEF administered a vetting survey to determine whether proposed villages qualified for the program. This survey, which we refer to as the baseline survey, was conducted in February 2009. Following the baseline survey, 263 villages that qualified for the program were randomly assigned to the two subsidy treatments, or to the control group. However, after random assignment, SEF scaled down the original evaluation sample of 263 villages to 199 villages to correct for errors in determining whether a village qualified for the program. The decisions taken by SEF were orthogonal to the assigned program status of the village. Thus, the effective evaluation sample consisted of 82 villages under the gender-uniform subsidy treatment, 79 under
the gender-differentiated one, and 38 as controls.
Schools were established in summer 2009. Because the new school year normally commences in the spring, program-school students had an abbreviated first school year. A follow-up survey was conducted in April/May 2011, after the conclusion of the second school year under the program.
The baseline survey consisted of a village survey answered by village leaders, a school survey of all schools in the general vicinity of the village, and a household survey of 12 households randomly selected from the list submitted by the entrepreneur of 75 households that had agreed to send their children to the proposed program school. The household survey collected information on the household, the household head, and on each child aged 5–9. In each village, the baseline survey also consisted of a survey of the entrepreneur and proposed teachers, as well as physical checks by the survey interviewers of the proposed school site and building. GPS data were collected from all schools, the proposed program-school site, and surveyed households.
The follow-up survey consisted of three instruments: a school survey; a household survey; and a child survey, which included a test. The household and child surveys, and child tests, were administered at the child’s home. The household survey was administered to households with at least one child aged 5–9. In large villages, up to 42 randomly sampled households in the village were interviewed; in villages with fewer than 42 households, which comprised the majority, all households in the village were interviewed. The survey was multi-topic, and had extensive modules on past and current schooling and other activities for children aged 5–17, answered by the household head or another primary adult household member.
A child survey was administered to each child aged 5–9. It asked questions mainly on work activities performed inside and outside the home, past and current schooling, and aspirations.
Each child was then administered tests on language (either Urdu or Sindhi, as preferred) and mathematics.
The school survey gathered information from interviews of head teachers and all other teachers, and visual checks by the survey interviewers of school infrastructural and environmental conditions. The survey also collected student attendance information through a headcount, with the attendance lists used during the household survey to verify the child’s enrollment status reported by the household. GPS data was gathered from all surveyed households and schools.