Back to History Current Version

Evaluating the Effect of the Primary Literacy Project on Literacy and Academic Achievement

Last registered on December 17, 2019

Pre-Trial

Trial Information

General Information

Title
Evaluating the Effect of the Primary Literacy Project on Literacy and Academic Achievement
RCT ID
AEARCTR-0000021
Initial registration date
November 20, 2013

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
November 20, 2013, 4:49 PM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
December 17, 2019, 5:25 PM EST

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Primary Investigator

Affiliation
University of Minnesota

Other Primary Investigator(s)

PI Affiliation
University of Michigan

Additional Trial Information

Status
On going
Start date
2013-02-04
End date
2021-12-31
Secondary IDs
Abstract
Despite ongoing efforts from both the Government of Uganda and the international donor community, many children in Uganda still are not acquiring literacy skills at a satisfactory level. This project evaluates an innovative new educational paradigm, the Primary Literacy Project (PLP), which currently focuses on P1 to P3 (first- to third-grade) students, employing a mother-tongue-first instructional approach and extensive teacher support and training. Using a randomized controlled trial this study measures the effects of the PLP on learning outcomes. We study 128 schools, which were randomly assigned to either the full PLP, a half-program that includes the materials from the PLP but where the training is provided through government-employed coordinating centre tutors (CCTs), or a control group. Outcomes are to be measured principally in terms of improvements in Early Grade Reading Assessment (EGRA) and Early Grade Writing Assessment (EGWA) scores. The results of this study will be used to further improve the PLP, helping to maximize both its academic benefits and cost-effectiveness; we also conduct classroom observations (using rubrics and/or videos), surveys of teachers and surveys of parents in order to capture additional outcomes.

This revision to the trial registration updates the study to account for a substantial expansion in its length and scope. The original study followed 38 schools for one year, 2013, focusing solely on P1. In the expanded study, we follow 128 schools (including the first 38) for up to five years, 2013 to 2017, following students from P1 all the way up to P3. We have an additional sample of 30 pure control schools originally sampled that we plan on collecting outcomes near the end of the study.
External Link(s)

Registration Citation

Citation
Kerwin, Jason and Rebecca Thornton. 2019. "Evaluating the Effect of the Primary Literacy Project on Literacy and Academic Achievement." AEA RCT Registry. December 17. https://doi.org/10.1257/rct.21-4.1
Former Citation
Kerwin, Jason and Rebecca Thornton. 2019. "Evaluating the Effect of the Primary Literacy Project on Literacy and Academic Achievement." AEA RCT Registry. December 17. https://www.socialscienceregistry.org/trials/21/history/59068
Experimental Details

Interventions

Intervention(s)
Intervention Start Date
2013-02-04
Intervention End Date
2016-12-02

Primary Outcomes

Primary Outcomes (end points)
Test scores, attendance, classroom observations, teacher/parent/child attitudes and behavior toward schooling, administrative records from schools (enrollment, marks, attendance)
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
Our experiment is an impact evaluation that randomly assigns schools to either receive the complete PLP intervention, partial PLP intervention or to receive the current standard educational serves, serving as a control.

A. Site and School Selection
Eligible schools will include primary schools from the Lango Sub-region. There are two sets of schools in the study, with slightly different selection criteria – Phase 1 schools and Phase 2 schools. Phase 1 schools were selected in late 2012 for participation in an initial phase of the RCT during the 2013 school year. Phase 2 schools were selected in late 2013 for participation in an expanded phase of the RCT from 2014 to 2016. The Phase 1 schools will also remain in the study through 2016.

• The Phase 1 schools had to meet the following criteria to be eligible for the study:
• Must be in Aculbanya, Adel, Adyel, Ayer, or Loro coordinating centre (CC)
• At least two P1 classes
• An early primary (P1 to P3) student-to-teacher ratio of 135 or less during the 2012 school year
• P1 classes must have desks
• 15 km or less from the CC
• A head teacher regarded as “engaged” by the CCT
• School must be accessible by road including during the rainy season
• Must not currently or previously be involved in Mango Tree’s project activities

There were 36 schools that met these criteria. This number was later expanded to 38: two schools were dropped and four more were added by relaxing the distance criterion to 20 km instead of 15.

• The Phase 2 schools had to meet the following criteria to be eligible for the study:
• Must be in the Lango Sub-region
• Must have desks in P1, P2 and P3 classrooms
• Must have blackboards in P1, P2 and P3 classrooms
• Must have 150 or fewer students per class in each of P1, P2 and P3 classrooms in the 2013 school year

There were 120 schools that met these criteria. Of these, 90 were selected (using stratified random selection) to participate in the study. 30 were selected to remain pure control schools for later evaluation of Hawthorne effects, or effects of the resources provided to the CCTs.

B. Randomization - Treatment and Control Schools and Classrooms
42 of the 128 schools are randomly assigned as controls, 44 schools as CCT treatment and another 42 schools assigned to the MT treatment. Beyond the data used to select the schools, additional school data – available from district education officers – including location in the Lango sub-region, average performance on previous national exams, the share of female students, and total P1 students, was used for stratified assignment of the treatment.

The random assignment of the treatment was stratified across the 128 target schools in order to ensure a similar distribution of school characteristics across the experimental arms. The 128 schools were divided into stratification groups that were as similar as possible; these sets of 3 schools are known as “stratification groups”. These groups were formed by matching schools on a list of school-level factors.
• For the Phase 1 schools, the stratification groups were formed by matching schools on CC, total P1 enrollment, and distance to the coordinating center (CC).
• For the Phase 2 schools, the stratification groups were formed by matching schools on CC, then on whether they had more than one P1 stream, then on whether they had smaller-than-median class sizes (less than 94.33 students/class from P1 to P3), then on whether they had a higher-than-median PLE pass rate (more than 4.47%), and then on whether they were closer than the median school to the CC (less than 9 km away).
Within each stratification group, treatment status was assigned at random at a public meeting of school officials - during December 2012 for the Phase 1 schools and during January 2014 for the Phase 2 schools. For every stratification group three colored tokens, one representing each study arm, were placed into an opaque bag and withdrawn one at a time by a neutral party in full view of all stakeholders. Different color tokens were used to represent the MT treatment, the CCT treatment, and the control group. The three schools were assigned to a specific order, with the first token drawn corresponding to the first school and so forth. Therefore the first token withdrawn determined the study arm assignment of the first school, the second token determined the assignment of the second school, and the third and final remaining token determined the assignment for the third school. This process yielded assignments of schools to study arms.

The schools randomly assigned to the MT treatment group receive the full LLP intervention, comprising materials, instructional videos, training by Mango Tree’s expert trainers, and classroom support supervision along with continuous professional development by expert trainers. The classrooms in the CCT treatment schools will receive the teaching and learning materials, including instructional videos. Their teachers will be trained and supported by the CCTs as part of their routine teacher training and support supervision duties. Beginning with the 2014 school year, the control schools also receive a set of wall charts covering topics unrelated to our main outcomes, as a gift to thank them for their participation in the study; we can evaluate any effects of these charts using the 30 pure control schools mentioned above.

For all classrooms, we will conduct both baseline and endline exams using the standardized EGRA and EGWA examinations as a measure of literacy competency. The table below shows the components of the LLP received by each study arm as well as the data to be collected from each arm. Wall clocks and slates were randomly allocated among schools in the CCT treatment and control arms. Among these, 20 received slates only, 23 received clocks only, 20 received both, and 23 received none. The control schools are affected both by the CCT training conducted by Mango Tree, and also by the data collection activities themselves. The set of 30 “pure” control schools will help us measure these potential effects.

In addition to the school-level randomizations of variants of the program and of materials, we will also randomize the provision, type, and content of report cards at the individual household level. The NULP provides report cards that are quite different from the typical report cards given out by Uganda’s Ministry of Education and Sports. We will randomize whether households in all three study arms receive regular report cards or NULP-style report cards, and also cross-randomize whether the report cards provide additional information on the returns to schooling.


C. Selection and Assignment of Teachers
• For the Phase 1 schools, the LLP intervention involved asking the head teacher for the school to choose the two best early primary teachers in the school and assign them to the two P1 classrooms. Typically these are the highest skilled teachers in the school who are in good health and are very committed to their work. In order to make each evaluation group equivalent, this was done in each of the 38 study schools for 2013. Head teachers agreed to assign the two best teachers to P1 classrooms, and to submit the names of those teachers at the stakeholder meeting prior to the public random assignment of the intervention. Baseline surveys of all teachers were conducted prior to the beginning of the school year. In addition, information on teacher attendance and performance was collected during random checks over the course of the school year. Compliance with this procedure was very high, and the data collection team noted all deviations from the pre-assignment of teachers to classrooms.

• For the Phase 2 schools, head teachers were asked to nominate teachers prior to learning their school’s treatment status by whatever process they normally use, and then provide their names to Mango Tree prior to the lottery that was done to determine which schools are assigned to which study arm.


D. Assignment of Students to Classrooms

One potential concern with the presence of two P1-P3 classrooms in each school is that the effect of an individual teacher may be hard to separate from that of the overall program. In particular, students may attempt to switch into the classroom with the stronger teacher, exaggerating the LLP’s effects. To mitigate this issue, students in schools with two (or more) classrooms will be randomly assigned to one of the two classrooms by the school’s head teacher. This will prevent any potential confounding from classroom-switching. It also helps ensure that the classroom assignments are fair, with each student having an equal chance of being paired with each teacher. This is particularly important in situations where more than two P1-P3 classrooms exist at a school.

This randomization will happen during two school years: 2013 and 2016. In 2013, the random assignment was conducted by using specialized enrollment rosters for the P1 classes in all 38 Phase 1 schools. Mango Tree staff carried pre-printed rosters with spaces for each students’ name and other details, along with an assignment to stream A or stream B. The assignments were generated at random using the runiform() function in Stata 11 SE. The head teachers were instructed to copy over the student names onto these rosters in order from their own records, and use the listed assignment for each student. Any late-enrolling students were added in the order they arrived. This process was validated by Mango Tree staff field visits, and a copy of each roster was collected and used to select students for examinations. We aim to follow the same procedure in 2016, for all 128 schools in the MT program, CCT program, and control groups, as long as the school has more than one classroom.
Experimental Design Details
Not available
Randomization Method
Public lottery, stratified on baseline school attributes including geography.
Randomization Unit
School
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
158 schools (30 are pure controls with no students sampled, except at the end of the study)

Number of students by cluster:
• Cohort 1 sample - 50 P1 students selected and tested in February 2014 at all 38 of the Phase 1 schools, stratified by gender.

• Cohort 2 sample -
- up to 100 P1 students were selected and tested in all 90 of the Phase 2 schools, stratified by gender.
- up to 100 P1 students were sampled in all 38 of the Phase 1 schools.

o In some cases, fewer than the target number of students were available at the school when we went in for testing
o The Cohort 2 sample was smaller for the 38 Phase 1 schools because we also tracked as many of the Cohort 1 students as possible from those schools, limiting the time available for testing new students.

• Cohorts 3, 4, and 5 – up to 100 P1 students will be selected and tested at each of the 128 schools
Sample size: planned number of observations
84,000 students
Sample size (or number of clusters) by treatment arms
Main Randomization: 42 schools full treatment, 44 schools half-treatment, 42 schools control, 30 schools pure control (no testing)
Cross-Randomization A: All full treatment schools research wall clocks and slates. Wall clocks and slates were randomly allocated among schools (in any grade in the sample) in the half-treatment and control arms. 20 received slates only, 23 received clocks only, 20 received both, and 23 received none.
Cross-Randomization B: Individual-level randomization of households (parents) to receive different types of report cards on their child’s performance in school. These report cards will be provided at school meetings. While the exact attendance at these meetings is hard to forecast, our best estimate is that an average of 70 parents per school to attend from each of the 128 schools in the full treatment, half-treatment, and control groups. This would yield a total sample size of roughly 9000.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Please see the separate Power Calculations document under Supporting Documents & Materials.
Supporting Documents and Materials

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
IRB

Institutional Review Boards (IRBs)

IRB Name
University of Michigan Institutional Review Board - Health Sciences and Behavioral Sciences
IRB Approval Date
2012-12-03
IRB Approval Number
HUM00070458
IRB Name
School of Biomedical Sciences Research and Ethics Committee, Makerere University
IRB Approval Date
2013-03-15
IRB Approval Number
SBS 063
Analysis Plan

Analysis Plan Documents

Analysis Plan for 2013 Data

MD5: 43fe4e6024912858a9ea06e2888108bd

SHA1: ce357b4bfb21c2055b42a71753b11258ce95122c

Uploaded At: November 20, 2013

PLP Evaluation Analysis Plan 2015-1-16

MD5: 8002abb0f1cecd167e69c345299233d1

SHA1: 13bd25391ff7c525c3c3527cbef4c16e88bd96e3

Uploaded At: January 16, 2015