Experimental Design Details
Although we expect to have over ~7,000 students in the experiment, logistical constraints do not allow for class- or individual-level randomization to maximize power. Even if this were logistically feasible, we worry that this intervention is particularly susceptible to frictions between called and uncalled students, or teachers assigned to make calls and those that were not. We are aware that the disadvantage of randomizing at the school-level is that power is severely harmed, and we try to maximize power by double-blocking on covariates which we hypothesize explain some of the variation in the outcome. First, we create three bins within the state by the population within a 5 km radius surrounding each school, as a proxy for urban/peri-urban/rural location. These bins span from ~6000 people to ~55,000 for the rural category, from ~55,000 to ~170,000 for the peri-urban category, and greater than ~170,000 (until ~1,850,000) for the urban category. The GIS population data comes from Bosco et al (2017), downloaded at a resolution of 1-km grids at the equator. Then, we split each bin into quintiles representing baseline exam-scores for each school. For each school, a weighted average z-score across the target grades was calculated based on the school’s math scores. Given our partner’s approach to testing and data collection, these scores are comparable across the state.
This blocking procedure leaves 15 randomization blocks, each with information about the type of location, and the baseline achievement level of each school. We randomly assign treatment to schools within each of these 15 blocks. All blocks have 7 schools within them. 10 of the blocks have 5 treated schools, and 5 blocks have 4 treated schools. Among the treated schools in each block, the treatment arm (“accountability” or “tutoring”) was then randomly assigned. Among the 5 blocks with 4 treated schools, exactly half of all treated schools are assigned to one of the treatment arms. Among the 10 blocks with 5 treatment schools, half of these blocks assign 3 schools to one treatment and 2 to the other treatment, and the other half of these blocks has the inverse assignment of treatment schools, for a highly balanced distribution of treatment assignment.