Back to History

Fields Changed

Registration

Field Before After
Trial Title Evaluating the Effect of the Primary Literacy Project on Literacy and Academic Achievement Evaluating the Effect of the Primary Literacy Project on Literacy and Academic Achievement
Abstract Despite ongoing efforts from both the Government of Uganda and the international donor community, many children in Uganda still are not acquiring literacy skills at a satisfactory level. This project evaluates an innovative new educational paradigm, the Primary Literacy Project (PLP), which focuses on P1 students, employing a mother-tongue-first instructional approach and extensive teacher support and training. Using a randomized controlled trial this study measures the effects of the PLP on learning outcomes. We study 38 schools, which were randomly assigned to either the full PLP, a half-program that includes the materials from the PLP but where the training is provided through government-employed coordinating centre tutors (CCTs), or a control group. Outcomes are to be measured principally in terms of improvements in Early Grade Reading Assessment (EGRA) and Early Grade Writing Assessment (EGWA) scores. The results of this study will be used to further improve the PLP, helping to maximize both its academic benefits and cost-effectiveness; in addition, we measure classroom observations, surveys of teachers and surveys of parents in order to capture additional outcomes. Despite ongoing efforts from both the Government of Uganda and the international donor community, many children in Uganda still are not acquiring literacy skills at a satisfactory level. This project evaluates an innovative new educational paradigm, the Primary Literacy Project (PLP), which currently focuses on P1 to P3 (first- to third-grade) students, employing a mother-tongue-first instructional approach and extensive teacher support and training. Using a randomized controlled trial this study measures the effects of the PLP on learning outcomes. We study 128 schools, which were randomly assigned to either the full PLP, a half-program that includes the materials from the PLP but where the training is provided through government-employed coordinating centre tutors (CCTs), or a control group. Outcomes are to be measured principally in terms of improvements in Early Grade Reading Assessment (EGRA) and Early Grade Writing Assessment (EGWA) scores. The results of this study will be used to further improve the PLP, helping to maximize both its academic benefits and cost-effectiveness; we also conduct classroom observations (using rubrics and/or videos), surveys of teachers and surveys of parents in order to capture additional outcomes. This revision to the trial registration updates the study to account for a substantial expansion in its length and scope. The original study followed 38 schools for one year, 2013, focusing solely on P1. In the expanded study, we follow 128 schools (including the first 38) for up to five years, 2013 to 2017, following students from P1 all the way up to P3. We have an additional sample of 30 pure control schools originally sampled that we plan on collecting outcomes near the end of the study.
Trial End Date November 29, 2013 December 02, 2017
Last Published November 20, 2013 04:49 PM January 16, 2015 05:56 PM
Intervention End Date November 29, 2013 December 02, 2016
Primary Outcomes (End Points) Test scores, attendance, classroom observations, teacher/parent/child attitudes and behavior toward schooling Test scores, attendance, classroom observations, teacher/parent/child attitudes and behavior toward schooling, administrative records from schools (enrollment, marks, attendance)
Experimental Design (Public) Our experiment is an impact evaluation that randomly assigned schools to either receive the complete PLP intervention, partial PLP intervention or to receive the current standard educational serves, serving as a control. A. Site and School Selection Eligible schools include primary schools from the Lango Sub-region that have two grade one classes (P1) and that agree to participate in the program. From this set of eligible schools, Mango Tree, in collaboration with the school administrations, district education officers and coordinating center tutors (CCTs), selected 36 target schools based on their location, availability of desks/tables in both the P1 classrooms, two P1 teachers for the two separate classrooms, willingness of the schools to sign a contract with Mango Tree and agree to place two best early primary teachers in the selected P1 classrooms. B. Randomization - Treatment and Control Schools and Classrooms 12 of the 36 schools were randomly assigned as controls, 12 schools as partial treatment and another 12 schools were assigned to the treatment. Beyond the data used to select the schools, additional school data – available from district education officers – including location in the Lango sub-region, average performance on previous national exams, the share of female pupils, and total P1 pupils, was used for stratified assignment of the treatment. The random assignment of the treatment was stratified across the 36 target schools in order to ensure a similar distribution of school characteristics across the control arm and the two treatment arms. C. Selection and Assignment of Teachers The PLP intervention involves asking the head teacher for the school to choose the two best early primary teachers in the school and assign them to the two P1 classrooms. Typically these are the highest skilled teachers in the school who are very committed to their work. In order to make each evaluation group equivalent, this was done in each of the 38 study schools. Head teachers agreed to assign the two best teachers to P1 classrooms, and to submit the names of those teachers at the stakeholder meeting prior to the public random assignment of the intervention. Baseline surveys of all P1 teachers were conducted at the beginning of the school. In addition, information on teacher attendance and performance will be collected during random checks over the course of the school year. Compliance with this procedure has thus far been very high, but any deviations from the pre-assignment of teachers to classrooms are being noted by the data collection team. A related issue is that one of the Control schools, Alyec, had only a single P1 Stream instead of the two that were agreed upon in the contract. This means that our sample of P1 teachers has been reduced by one, to 75. It may also affect the validity of comparisons between Alyec and other schools in our sample, as large class sizes have a detrimental effect on learning; the single P1 stream at Alyec had over 200 pupils. Our analyses will take this potential issue into account, and Mango Tree staff are working with Alyec to correct the problem. Any other compliance issues will be noted as they come up, and either mitigated on the ground or accounted for in our data analyses. D. Assignment of Pupils to Classrooms Students were randomly assigned to one of the two classrooms by the school’s head teacher. This prevents any potential confounding from classroom-switching. It also helps ensure that the classroom assignments are fair, with each pupil having an equal chance of being paired with each teacher. The random assignment was conducted by using specialized enrollment rosters for all 38 schools. Mango Tree staff carried pre-printed rosters with spaces for each pupils’ name and other details, along with an assignment to stream A or stream B. The assignments were generated at random using the runiform() function in Stata 11 SE. The head teachers were instructed to copy over the pupil names onto these rosters in order from their own records, and use the listed assignment for each pupil. Any late-enrolling pupils were added in the order they arrived. This process was validated by Mango Tree staff field visits, and a copy of each roster was collected and used to select pupils for examinations. Our experiment is an impact evaluation that randomly assigns schools to either receive the complete PLP intervention, partial PLP intervention or to receive the current standard educational serves, serving as a control. A. Site and School Selection Eligible schools will include primary schools from the Lango Sub-region. There are two sets of schools in the study, with slightly different selection criteria – Phase 1 schools and Phase 2 schools. Phase 1 schools were selected in late 2012 for participation in an initial phase of the RCT during the 2013 school year. Phase 2 schools were selected in late 2013 for participation in an expanded phase of the RCT from 2014 to 2016. The Phase 1 schools will also remain in the study through 2016. • The Phase 1 schools had to meet the following criteria to be eligible for the study: • Must be in Aculbanya, Adel, Adyel, Ayer, or Loro coordinating centre (CC) • At least two P1 classes • An early primary (P1 to P3) student-to-teacher ratio of 135 or less during the 2012 school year • P1 classes must have desks • 15 km or less from the CC • A head teacher regarded as “engaged” by the CCT • School must be accessible by road including during the rainy season • Must not currently or previously be involved in Mango Tree’s project activities There were 36 schools that met these criteria. This number was later expanded to 38: two schools were dropped and four more were added by relaxing the distance criterion to 20 km instead of 15. • The Phase 2 schools had to meet the following criteria to be eligible for the study: • Must be in the Lango Sub-region • Must have desks in P1, P2 and P3 classrooms • Must have blackboards in P1, P2 and P3 classrooms • Must have 150 or fewer students per class in each of P1, P2 and P3 classrooms in the 2013 school year There were 120 schools that met these criteria. Of these, 90 were selected (using stratified random selection) to participate in the study. 30 were selected to remain pure control schools for later evaluation of Hawthorne effects, or effects of the resources provided to the CCTs. B. Randomization - Treatment and Control Schools and Classrooms 42 of the 128 schools are randomly assigned as controls, 44 schools as CCT treatment and another 42 schools assigned to the MT treatment. Beyond the data used to select the schools, additional school data – available from district education officers – including location in the Lango sub-region, average performance on previous national exams, the share of female students, and total P1 students, was used for stratified assignment of the treatment. The random assignment of the treatment was stratified across the 128 target schools in order to ensure a similar distribution of school characteristics across the experimental arms. The 128 schools were divided into stratification groups that were as similar as possible; these sets of 3 schools are known as “stratification groups”. These groups were formed by matching schools on a list of school-level factors. • For the Phase 1 schools, the stratification groups were formed by matching schools on CC, total P1 enrollment, and distance to the coordinating center (CC). • For the Phase 2 schools, the stratification groups were formed by matching schools on CC, then on whether they had more than one P1 stream, then on whether they had smaller-than-median class sizes (less than 94.33 students/class from P1 to P3), then on whether they had a higher-than-median PLE pass rate (more than 4.47%), and then on whether they were closer than the median school to the CC (less than 9 km away). Within each stratification group, treatment status was assigned at random at a public meeting of school officials - during December 2012 for the Phase 1 schools and during January 2014 for the Phase 2 schools. For every stratification group three colored tokens, one representing each study arm, were placed into an opaque bag and withdrawn one at a time by a neutral party in full view of all stakeholders. Different color tokens were used to represent the MT treatment, the CCT treatment, and the control group. The three schools were assigned to a specific order, with the first token drawn corresponding to the first school and so forth. Therefore the first token withdrawn determined the study arm assignment of the first school, the second token determined the assignment of the second school, and the third and final remaining token determined the assignment for the third school. This process yielded assignments of schools to study arms. The schools randomly assigned to the MT treatment group receive the full LLP intervention, comprising materials, instructional videos, training by Mango Tree’s expert trainers, and classroom support supervision along with continuous professional development by expert trainers. The classrooms in the CCT treatment schools will receive the teaching and learning materials, including instructional videos. Their teachers will be trained and supported by the CCTs as part of their routine teacher training and support supervision duties. Beginning with the 2014 school year, the control schools also receive a set of wall charts covering topics unrelated to our main outcomes, as a gift to thank them for their participation in the study; we can evaluate any effects of these charts using the 30 pure control schools mentioned above. For all classrooms, we will conduct both baseline and endline exams using the standardized EGRA and EGWA examinations as a measure of literacy competency. The table below shows the components of the LLP received by each study arm as well as the data to be collected from each arm. Wall clocks and slates were randomly allocated among schools in the CCT treatment and control arms. Among these, 20 received slates only, 23 received clocks only, 20 received both, and 23 received none. The control schools are affected both by the CCT training conducted by Mango Tree, and also by the data collection activities themselves. The set of 30 “pure” control schools will help us measure these potential effects. In addition to the school-level randomizations of variants of the program and of materials, we will also randomize the provision, type, and content of report cards at the individual household level. The NULP provides report cards that are quite different from the typical report cards given out by Uganda’s Ministry of Education and Sports. We will randomize whether households in all three study arms receive regular report cards or NULP-style report cards, and also cross-randomize whether the report cards provide additional information on the returns to schooling. C. Selection and Assignment of Teachers • For the Phase 1 schools, the LLP intervention involved asking the head teacher for the school to choose the two best early primary teachers in the school and assign them to the two P1 classrooms. Typically these are the highest skilled teachers in the school who are in good health and are very committed to their work. In order to make each evaluation group equivalent, this was done in each of the 38 study schools for 2013. Head teachers agreed to assign the two best teachers to P1 classrooms, and to submit the names of those teachers at the stakeholder meeting prior to the public random assignment of the intervention. Baseline surveys of all teachers were conducted prior to the beginning of the school year. In addition, information on teacher attendance and performance was collected during random checks over the course of the school year. Compliance with this procedure was very high, and the data collection team noted all deviations from the pre-assignment of teachers to classrooms. • For the Phase 2 schools, head teachers were asked to nominate teachers prior to learning their school’s treatment status by whatever process they normally use, and then provide their names to Mango Tree prior to the lottery that was done to determine which schools are assigned to which study arm. D. Assignment of Students to Classrooms One potential concern with the presence of two P1-P3 classrooms in each school is that the effect of an individual teacher may be hard to separate from that of the overall program. In particular, students may attempt to switch into the classroom with the stronger teacher, exaggerating the LLP’s effects. To mitigate this issue, students in schools with two (or more) classrooms will be randomly assigned to one of the two classrooms by the school’s head teacher. This will prevent any potential confounding from classroom-switching. It also helps ensure that the classroom assignments are fair, with each student having an equal chance of being paired with each teacher. This is particularly important in situations where more than two P1-P3 classrooms exist at a school. This randomization will happen during two school years: 2013 and 2016. In 2013, the random assignment was conducted by using specialized enrollment rosters for the P1 classes in all 38 Phase 1 schools. Mango Tree staff carried pre-printed rosters with spaces for each students’ name and other details, along with an assignment to stream A or stream B. The assignments were generated at random using the runiform() function in Stata 11 SE. The head teachers were instructed to copy over the student names onto these rosters in order from their own records, and use the listed assignment for each student. Any late-enrolling students were added in the order they arrived. This process was validated by Mango Tree staff field visits, and a copy of each roster was collected and used to select students for examinations. We aim to follow the same procedure in 2016, for all 128 schools in the MT program, CCT program, and control groups, as long as the school has more than one classroom.
Planned Number of Clusters 38 schools 158 schools (30 are pure controls with no students sampled, except at the end of the study) Number of students by cluster: • Cohort 1 sample - 50 P1 students selected and tested in February 2014 at all 38 of the Phase 1 schools, stratified by gender. • Cohort 2 sample - - up to 100 P1 students were selected and tested in all 90 of the Phase 2 schools, stratified by gender. - up to 100 P1 students were sampled in all 38 of the Phase 1 schools. o In some cases, fewer than the target number of students were available at the school when we went in for testing o The Cohort 2 sample was smaller for the 38 Phase 1 schools because we also tracked as many of the Cohort 1 students as possible from those schools, limiting the time available for testing new students. • Cohorts 3, 4, and 5 – up to 100 P1 students will be selected and tested at each of the 128 schools
Planned Number of Observations 1900 pupils 84,000 students
Sample size (or number of clusters) by treatment arms 12 schools full treatment, 14 schools half-treatment, 12 schools control Main Randomization: 42 schools full treatment, 44 schools half-treatment, 42 schools control, 30 schools pure control (no testing) Cross-Randomization A: All full treatment schools research wall clocks and slates. Wall clocks and slates were randomly allocated among schools (in any grade in the sample) in the half-treatment and control arms. 20 received slates only, 23 received clocks only, 20 received both, and 23 received none. Cross-Randomization B: Individual-level randomization of households (parents) to receive different types of report cards on their child’s performance in school. These report cards will be provided at school meetings. While the exact attendance at these meetings is hard to forecast, our best estimate is that an average of 70 parents per school to attend from each of the 128 schools in the full treatment, half-treatment, and control groups. This would yield a total sample size of roughly 9000.
Power calculation: Minimum Detectable Effect Size for Main Outcomes 0.35 SDs of endline literacy test scores (intra-class correlation=0.2, partial R-squared of baseline scores = 0.7, power = 0.8, alpha = 0.05) Please see the separate Power Calculations document under Supporting Documents & Materials.
First registered on November 20, 2013
Back to top