Improving pedagogy at-scale in low-income countries

Interventions(s): This study will consist of two different RCTs modifying pedagogical practices, by introducing “spaced retrieval” in grade 3 science lessons, and “interleaving” in math lessons for grades 4-6. Schools will be randomly assigned to being part of the treatment group, or the comparison group, and will be part of that experimental group for both trials. Below, we describe the specific interventions in greater detail:

Spaced Retrieval in Grade 3 Science
This intervention aims to give a daily dosage of “mixed review” questions to students in Grade 3 Science, and to ensure that they are given an opportunity first to attempt to answer the question without any support from the book. It will give them a chance to find the answer in their text. After that, the teacher will share the correct answer. “Mixed Review” activities will vary between full lesson length of 45 minutes (12 lessons), 15 minutes (14 lessons), and 5 minutes (35 lessons), with the number of questions varying proportionally according to how much time is allotted for the review.
Interleaving in Upper Primary Mathematics
The current status quo of the math lessons is that each lesson has 20 practice problems, and all of those problems are blocked practice (i.e. aligned to the content of that lesson). This intervention would include changing those practice problems from blocked to interleaved. 40% of the practice problems would remain focused on the topic taught that day, while 60% of the practice problems would be distributed among the past 4 “levels” of content. This technique creates an opportunity for retrieving the interleaved content, but it also creates what cognitive scientists call “desirable difficulty”: the brain retains new information much better and retains it for longer if it meets a certain threshold of rigor (when the difficulties are palpable but learners have the means to overcome them).

2022-04-11

2022-09-30

Standardized school grades for Math and Science at the end of the term (~18 weeks after the expected start)

The outcomes for test scores, and enrollment come from our administrative data. The internal data collection process is mostly standardized within our network, allowing us to make direct comparisons for most outcomes and indicators across contexts. Similarly, within each context, all children take the same test within a testing round (i.e., during marking period tests and semester exams), allowing for comparisons across schools within each context.

For the trial on “spaced retrieval”, we plan to use Science outcomes as the main set of outcomes. For the interleaving trial, we plan to use Math scores, both standardized by grade, and pooled within a single regression but with grade fixed effects.

There is a possibility that we will get access to item-level data for one or both trials. If that is the case for all schools and pupils, we will present results using two types of outcomes: 1) with the outcome variable being the percentage of questions answered correctly, and 2) an IRT (2 parameter) score for subject*pupil (models run by grade). We will also report the correlation between these two types of scores, and if that is higher than 95%, we will show only one set of results. If we do not have access to item-level data, we will simply use the percentage of questions answered correctly as the outcome.

To get at the intent-to-treat effects (ITT), we will run a linear model with each outcome regressed on an indicator for whether each school was treated, along with fixed-effects for each randomization strata and the clustering of standard errors by school. Beyond the ITT, we also plan to study the treatment-on-the-treated effect (TOT). To do so, we will classify each school as “complier” or “not complier”. We will then define “complier” in two ways: 1) average lesson completion rate for the targeted subject in each trail above 50%, above 75%, above 90%, and 2) average lesson completion rate above the sample median, and the percentiles 75 and 90. The treated group will be then the set of schools in the treatment group, who were compliers given the categories above.

Finally, we will also carry out sub-group analyses by class size, school enrollment, baseline student performance, school location, lesson completion rates, and teacher and pupil attendance rates.

-Standardized school grades for other subjects not in the primary set of outcomes at the end of the term (~18 weeks after the expected start)
-Pupil and teacher attendance
-Student dropout and enrollment
-Lesson completion rates

To minimize potential spillovers of the treatment, we randomly allocate treatment at the school level. To increase statistical power, we create randomization strata that absorb some of the variation in outcomes. In particular, we create 8 strata based on previous academic performance for schools for which we have that data. For those schools that we do not have baseline assessment data, we stratify in 4 groups based on pupil attendance. For those schools with neither set of baseline data, we pool them into a single strata. Finally, we leave in a single strata 14 schools which we know that have had operational issues at the start of the semester, and as such may not be able to implement the intervention. We reserve the right to exclude them from the analysis, if the qualitative reports from the field point in the direction of implementation not being successful in the strata. In total, we are left with 14 different strata, within which treatment was randomly assigned to approximately half of all schools.

Randomization performed using Stata 17 by researchers.

Schools

Yes

312 schools

Given that the average classroom across the 312 schools, has 27.4 pupils, and there is on average one stream per grade, we expect to have 25,646 pupils for the interleaving trial, and 8,549 for the spaced retrieval trial.

Half of all schools are assigned to treatment, so we expect to have 156 schools in the treatment group.

For the spaced retrieval trial, we expect to have that having on average 27 students per school across all 312 schools in the sample, with 50% assigned to treatment, assuming an R2 explained by individual and school-level covariates to be 0.45, no-show rate (or missingness rate) of 30% and intra-class correlations of 0.20, consistent with our previous internal analytical work, we would expect a minimum-detectable effect (MDE) of 0.16 standard deviations (SD). We use the same parameters for the trial on interleaving, except that the number of students per school is 81, as this trial covers three grades. In this case, the MDE is 0.15 SDs.