Primary Outcomes (explanation)
The outcomes for test scores, and enrollment come from our administrative data. The internal data collection process is mostly standardized within our network, allowing us to make direct comparisons for most outcomes and indicators across contexts. Similarly, within each context, all children take the same test within a testing round (i.e., during marking period tests and semester exams), allowing for comparisons across schools within each context.
For the trial on “spaced retrieval”, we plan to use Science outcomes as the main set of outcomes. For the interleaving trial, we plan to use Math scores, both standardized by grade, and pooled within a single regression but with grade fixed effects.
There is a possibility that we will get access to item-level data for one or both trials. If that is the case for all schools and pupils, we will present results using two types of outcomes: 1) with the outcome variable being the percentage of questions answered correctly, and 2) an IRT (2 parameter) score for subject*pupil (models run by grade). We will also report the correlation between these two types of scores, and if that is higher than 95%, we will show only one set of results. If we do not have access to item-level data, we will simply use the percentage of questions answered correctly as the outcome.
To get at the intent-to-treat effects (ITT), we will run a linear model with each outcome regressed on an indicator for whether each school was treated, along with fixed-effects for each randomization strata and the clustering of standard errors by school. Beyond the ITT, we also plan to study the treatment-on-the-treated effect (TOT). To do so, we will classify each school as “complier” or “not complier”. We will then define “complier” in two ways: 1) average lesson completion rate for the targeted subject in each trail above 50%, above 75%, above 90%, and 2) average lesson completion rate above the sample median, and the percentiles 75 and 90. The treated group will be then the set of schools in the treatment group, who were compliers given the categories above.
Finally, we will also carry out sub-group analyses by class size, school enrollment, baseline student performance, school location, lesson completion rates, and teacher and pupil attendance rates.