Experimental Design
Our experiment is an impact evaluation that randomly assigns schools to either receive the complete PLP intervention, partial PLP intervention or to receive the current standard educational serves, serving as a control.
A. Site and School Selection
Eligible schools will include primary schools from the Lango Sub-region. There are two sets of schools in the study, with slightly different selection criteria – Phase 1 schools and Phase 2 schools. Phase 1 schools were selected in late 2012 for participation in an initial phase of the RCT during the 2013 school year. Phase 2 schools were selected in late 2013 for participation in an expanded phase of the RCT from 2014 to 2016. The Phase 1 schools will also remain in the study through 2016.
• The Phase 1 schools had to meet the following criteria to be eligible for the study:
• Must be in Aculbanya, Adel, Adyel, Ayer, or Loro coordinating centre (CC)
• At least two P1 classes
• An early primary (P1 to P3) student-to-teacher ratio of 135 or less during the 2012 school year
• P1 classes must have desks
• 15 km or less from the CC
• A head teacher regarded as “engaged” by the CCT
• School must be accessible by road including during the rainy season
• Must not currently or previously be involved in Mango Tree’s project activities
There were 36 schools that met these criteria. This number was later expanded to 38: two schools were dropped and four more were added by relaxing the distance criterion to 20 km instead of 15.
• The Phase 2 schools had to meet the following criteria to be eligible for the study:
• Must be in the Lango Sub-region
• Must have desks in P1, P2 and P3 classrooms
• Must have blackboards in P1, P2 and P3 classrooms
• Must have 150 or fewer students per class in each of P1, P2 and P3 classrooms in the 2013 school year
There were 120 schools that met these criteria. Of these, 90 were selected (using stratified random selection) to participate in the study. 30 were selected to remain pure control schools for later evaluation of Hawthorne effects, or effects of the resources provided to the CCTs.
B. Randomization - Treatment and Control Schools and Classrooms
42 of the 128 schools are randomly assigned as controls, 44 schools as CCT treatment and another 42 schools assigned to the MT treatment. Beyond the data used to select the schools, additional school data – available from district education officers – including location in the Lango sub-region, average performance on previous national exams, the share of female students, and total P1 students, was used for stratified assignment of the treatment.
The random assignment of the treatment was stratified across the 128 target schools in order to ensure a similar distribution of school characteristics across the experimental arms. The 128 schools were divided into stratification groups that were as similar as possible; these sets of 3 schools are known as “stratification groups”. These groups were formed by matching schools on a list of school-level factors.
• For the Phase 1 schools, the stratification groups were formed by matching schools on CC, total P1 enrollment, and distance to the coordinating center (CC).
• For the Phase 2 schools, the stratification groups were formed by matching schools on CC, then on whether they had more than one P1 stream, then on whether they had smaller-than-median class sizes (less than 94.33 students/class from P1 to P3), then on whether they had a higher-than-median PLE pass rate (more than 4.47%), and then on whether they were closer than the median school to the CC (less than 9 km away).
Within each stratification group, treatment status was assigned at random at a public meeting of school officials - during December 2012 for the Phase 1 schools and during January 2014 for the Phase 2 schools. For every stratification group three colored tokens, one representing each study arm, were placed into an opaque bag and withdrawn one at a time by a neutral party in full view of all stakeholders. Different color tokens were used to represent the MT treatment, the CCT treatment, and the control group. The three schools were assigned to a specific order, with the first token drawn corresponding to the first school and so forth. Therefore the first token withdrawn determined the study arm assignment of the first school, the second token determined the assignment of the second school, and the third and final remaining token determined the assignment for the third school. This process yielded assignments of schools to study arms.
The schools randomly assigned to the MT treatment group receive the full NULP intervention, comprising materials, instructional videos, training by Mango Tree’s expert trainers, and classroom support supervision along with continuous professional development by expert trainers. The classrooms in the CCT treatment schools will receive the teaching and learning materials, including instructional videos. Their teachers will be trained and supported by the CCTs as part of their routine teacher training and support supervision duties. Beginning with the 2014 school year, the control schools also receive a set of wall charts covering topics unrelated to our main outcomes, as a gift to thank them for their participation in the study; we can evaluate any effects of these charts using the 30 pure control schools mentioned above.
For all classrooms, we will conduct both baseline and endline exams using the standardized EGRA and EGWA examinations as a measure of literacy competency. The table below shows the components of the NULP received by each study arm as well as the data to be collected from each arm. Wall clocks and slates were randomly allocated among schools in the CCT treatment and control arms. Among these, 20 received slates only, 23 received clocks only, 20 received both, and 23 received none. The control schools are affected both by the CCT training conducted by Mango Tree, and also by the data collection activities themselves. The set of 30 “pure” control schools will help us measure these potential effects.
In addition to the school-level randomizations of variants of the program and of materials, we will also randomize the provision, type, and content of report cards at the individual household level. The NULP provides report cards that are quite different from the typical report cards given out by Uganda’s Ministry of Education and Sports. We will randomize whether households in all three study arms receive regular report cards or NULP-style report cards, and also cross-randomize whether the report cards provide additional information on the returns to schooling.
C. Selection and Assignment of Teachers
• For the Phase 1 schools, the NULP intervention involved asking the head teacher for the school to choose the two best early primary teachers in the school and assign them to the two P1 classrooms. Typically these are the highest skilled teachers in the school who are in good health and are very committed to their work. In order to make each evaluation group equivalent, this was done in each of the 38 study schools for 2013. Head teachers agreed to assign the two best teachers to P1 classrooms, and to submit the names of those teachers at the stakeholder meeting prior to the public random assignment of the intervention. Baseline surveys of all teachers were conducted prior to the beginning of the school year. In addition, information on teacher attendance and performance was collected during random checks over the course of the school year. Compliance with this procedure was very high, and the data collection team noted all deviations from the pre-assignment of teachers to classrooms.
• For the Phase 2 schools, head teachers were asked to nominate teachers prior to learning their school’s treatment status by whatever process they normally use, and then provide their names to Mango Tree prior to the lottery that was done to determine which schools are assigned to which study arm.
D. Assignment of Students to Classrooms
One potential concern with the presence of two P1-P3 classrooms in each school is that the effect of an individual teacher may be hard to separate from that of the overall program. In particular, students may attempt to switch into the classroom with the stronger teacher, exaggerating the NULP’s effects. To mitigate this issue, students in schools with two (or more) classrooms will be randomly assigned to one of the two classrooms by the school’s head teacher. This will prevent any potential confounding from classroom-switching. It also helps ensure that the classroom assignments are fair, with each student having an equal chance of being paired with each teacher. This is particularly important in situations where more than two P1-P3 classrooms exist at a school.
This randomization will happen during two school years: 2013 and 2016. In 2013, the random assignment was conducted by using specialized enrollment rosters for the P1 classes in all 38 Phase 1 schools. Mango Tree staff carried pre-printed rosters with spaces for each students’ name and other details, along with an assignment to stream A or stream B. The assignments were generated at random using the runiform() function in Stata 11 SE. The head teachers were instructed to copy over the student names onto these rosters in order from their own records, and use the listed assignment for each student. Any late-enrolling students were added in the order they arrived. This process was validated by Mango Tree staff field visits, and a copy of each roster was collected and used to select students for examinations. We aim to follow the same procedure in 2016, for all 128 schools in the MT program, CCT program, and control groups, as long as the school has more than one classroom.