Sustainability Impact Evaluation of the Early Grade Reading Study (EGRS I) and Impact Evaluation of the Reading Support Project (RSP)

DESCRIPTION OF THE EGRS I
This EGRS I was set up as a randomized control trial. The EGRS included three different interventions, all aimed at improving early-grade reading in Setswana. These three interventions were implemented with the teachers of a cohort of learners in Grade 1 in 2015, the teachers of the same cohort of learners in Grade 2 in 2016, and the first two interventions were extended to the teachers of the same learners again in Grade 3 in 2017.
Treatment 1: Training, scripted lessons, graded readers
Treatments 1 and 2 aimed to apply the same set of instructional practices in the teaching of home language literacy in grade 1, 2 and 3 classrooms working in one grade at a time. Both treatments therefore provided teachers with clearly scripted lesson plans, which were aligned to the curriculum as specified in the Curriculum and Assessment Policy Statements (CAPS) for home language literacy in the Foundation Phase. The lesson plans incorporated the use of learning support materials including the government-provided workbooks as well as certain additional materials (graded reading booklets, flash cards, posters, etc.), which were provided through the EGRS. The graded reading booklets provided a key resource for the teacher to use in group-guided reading and individual work so as to facilitate reading practice at an appropriate pace and sequence of progression.
Treatment 1 trained the teachers on how to use the lesson plans and accompanying materials through central training sessions, each lasting two days, and occurring twice yearly.
Treatment 2: Reading Coaches, scripted lessons, graded readers
Exactly the same set of instructional materials (scripted lesson plans, graded reading booklets and other materials) were provided to Treatment 2 schools. However, instead of central training sessions, one day training/orientation was provided at the start of each term, accompanied by ongoing support to teachers consisting of regular (monthly) on-school coaching from specialist “reading coaches” visits. In addition to these on-site visits, there were occasional needs-based workshops with the coach and a small cluster of nearby Treatment 2 schools.
Treatment 3: Parental involvement
Treatment 3 was designed to promote parental involvement to support their children’s reading progress. At each of the 50 schools in this treatment arm a Community Reading Coach (CRC) was recruited. The CRC was identified through communication with the school principal who recommended a suitably qualified, but available person in the community. The CRCs attended a one-day training session facilitated by the service provider (Class Act) at the start of each school term (quarterly). The CRCs were trained to deliver weekly training sessions for Grade 1 parents at their respective schools. A total of 30 sessions were scheduled for each year covering a total of 10 topics. The parental involvement intervention arm was prematurely ended in 2016, after two years of implementation working first with Grade 1 parents and then Grade 2 parents. The parents of Grade 3 learners were not part of the intervention.
DESCRIPTION OF THE RSP
The RSP was set up as a randomized control trial, with six focus areas.
1. Quarterly ‘just in time’ training for teachers (from 263 schools) on the implementations of Curriculum and Assessment Policy Statement (CAPS) and lesson plans (all 263 schools)
2. The provision of Learning and Teaching Support Material (LTSM) packages (all 263 schools).
3. The provision of 14 literacy coaches to 140 schools to offer classroom-based support to Foundation Phase teachers (Coaching in 140 schools)
4. The development of the leadership capacity of principals/deputies and Head of Departments (HODs) to promote a culture of reading in their schools (SMT training in 65 schools)
5. Through the DBE, Classroom libraries are provided to 98 schools
The RSP specifically aims to: improve subject matter knowledge; promote more effective pedagogic practices; improve in-class time management; increase effective use of LTSM; and foster a school environment to support teachers’ ability to implement the full curriculum and facilitate successful teaching and learning.
INTERACTIONS BETWEEN EGRS I AND RSP
In order to set up research questions and analyses accurately, it is helpful to understand how the EGRS and RSP interventions may interact and how they compare, in three areas.
1) Teacher Training
Teacher training for EGRS I focused on one language (Setswana) and trained one grade of teachers each year, for one year (Grade 1, then Grade 2, then Grade 3). RSP trained all three groups of teachers at the same time for two years in both Setswana and English. We may expect interactions between the two interventions in that some teachers may have received training in Setswana instructions under both programs.
2) Coaching
Coaches used in the EGRS program were not reemployed under RSP. Thus, we expect interactions to be limited in the coaching intervention of the two programs. It is possible that teachers who received coaching under EGRS may have been more experienced in getting feedback and may have benefited more from the RSP coaching as a result.
3) Learner and Teacher Support Materials
These materials were provided under both programs. In EGRS they were only provided for Setswana while the availability of EFAL materials to teachers was not provided and it is not clear what materials were used. In RSP materials were provided for both Setswana and EFAL. These included structured lesson plans, readers and other supporting LTSM. It is possible that the EGRS teachers were better prepared for LTSM use and further leveraged this when receiving RSP materials for both languages.
 
3.2. PRIMARY OUTCOMES
The primary outcome of interest is reading proficiency in Setswana home language (HL) and English first additional language (EFAL). For reading proficiency we will measure all the intermediate steps towards comprehension: letter recognition, phonetic awareness, word recognition and reading, vocabulary, reading fluency, and reading with comprehension. These tests are adapted from standard tests that have already been developed for the English First Additional language and Setswana language, such as the Early-Grade-Reading-Assessment (EGRA), but also newly developed and extensively piloted reading tasks. As our main outcome indicator, we will construct an aggregate indicator of learning proficiency, using principal component analysis.

2015-05-01

2020-12-31

The primary outcome of interest is reading proficiency in Setswana home language (HL) and English first additional language (EFAL). For reading proficiency we will measure all the intermediate steps towards comprehension: letter recognition, phonetic awareness, word recognition and reading, vocabulary, reading fluency, and reading with comprehension. These tests are adapted from standard tests that have already been developed for the English First Additional language and Setswana language, such as the Early-Grade-Reading-Assessment (EGRA), but also newly developed and extensively piloted reading tasks. As our main outcome indicator, we will construct an aggregate indicator of learning proficiency, using principal component analysis.

EGRS
What is the impact of (i) three years of Coaching + TT + LTSM (ii) three years of TT + LTSM and (iii) two years of Parental Involvement on reading outcomes in Setswana and EFAL, compared to a comparison group receiving no intervention, nearly four years after the completion of all interventions.
RSP
What is the impact of (i) SMT + Coaching + TT + LTSM and (ii) Coaching + TT + LTSM compared to a comparison group receiving TT + LTSM alone on:
• Reading outcomes in Grade 2 and 3 in Setswana and EFAL?
• Classroom processes, teacher behavior, school functionality?
Within the main experimental groups, what is the impact of the Classroom Library Program on Reading outcomes in Grades 2 and 3 in Setswana and EFAL?
What is the relationship between fidelity of implementation (i.e. attendance at teacher and SMT training, participation in individual and group coaching) and the improvement in reading in the three groups?
What are the school-level factors mediating the effect of each intervention on learning outcomes?
How do treatment effects differ between subgroups based on, for example, pupil gender, teacher experience, school location and household profile in parent/guardian education, employment, literacy practices.?
What is the impact of:
i) 5 years of coaching (EGRS 3 years + RSP 2 years) + 5 years of TT + 5 years of LTSM
ii) 2 years of RSP coaching + 5 years of TT + 5 years of LTSM
iii) 2 years of parental involvement + 2 years of RSP coaching +2 years of RSP TT+ 2 years of RSP LTSM
The skills to be assessed include
Grade 3 learners:
• Home language: Rapid object naming, letter sound recognition, complex consonants & diacritics recognition, word reading, oral reading fluency and comprehension and listening comprehension.
• EFAL: Decodable word reading, sight word reading, oral reading fluency and comprehension.
Grade 4 learners:
• Home language: Rapid object naming, letter sound recognition, complex consonants & diacritics recognition, word reading, oral reading fluency and comprehension, written text comprehension.
• EFAL: Decodable word reading, oral reading fluency and comprehension, written text comprehension.
Grade 7 learners:
• Home language: Oral reading fluency and comprehension, written text comprehension, vocabulary.
• EFAL: Oral reading fluency and comprehension, written text comprehension, vocabulary.
The instruments include tasks that were adapted from standard early grade reading assessments in Setswana and English firs additional language, as well as newly developed and extensively piloted reading tasks.
As our main outcome indicator, we will construct an aggregate indicator of HL and EFAL learning proficiency, using principal component analysis.

For the evaluation of RSP we will assess the role of intermediate outcomes in achieving impact in two categories – fidelity of implementation and mediators.
Fidelity of Implementation
The fidelity of implementation analysis will aim to understand whether schools implementing the interventions to a higher standard achieved greater improvements in reading outcomes. To do this, we will identify candidate variables from the contextual tools during the data collection pilot, in addition to data already collected from the RSP implementation to indicate fidelity of implementation (e.g., teacher use of national workbook) and assess the relationship between each of these variables and reading outcomes. It is not possible to identify a causal relationship between fidelity of implementation and reading outcomes because of confounding factors. To control for observed confounding factors, we will include baseline characteristics of schools as covariates in these analyses.
Data available from the RSP implementation evaluation include: number of teachers per school attending training and receiving coaching and whether schools received project materials.
Mediators
The mediator analysis aims to identify the mechanism for the impact of interventions on outcomes. Mediators are typically school level processes (e.g., teacher corrects learner work) that are enhanced by the intervention and which lead to improved reading outcomes. Mediator variables that meet some or all of the following criterial will be considered.
(1) The intervention has a significant impact on the mediator
(2) There is a significant relationship between the mediator and reading outcomes
(3) The impact of the treatment variables on HL and EFAL reading proficiency is attenuated when the mediator is included in the regression equation.
Below are potential mediating variables (these will be determined during the final data analysis).
Teachers
Did the EGRS I and RSP treatment arms which involves training and coaching change teaching practice and effort?
1. Provide regular individualized assessment [TRQ – 3.7 and 3.10, CLW Section 2
document review)]
2. Require student to read out loud [TRQ – 3.7 and 3.10)]
3. Stream by ability [TRQ – 3.5]
4. Assess reading ability [TRQ – 3.7 and 3.10]
5. The completion of more writing exercises [TRQ – 3.7 and 3.10, CLW Section 2
document review]
6. The assessment of more writing exercises [CLW Section 2 document review]
7. Use of the national workbook [CLW Section 2 document review]
8. Increase the availability of home language text in classrooms [CLW Section 3]

Parents
Did EGRS I treatment arm lead to a change in parent attitudes and behavior?
1. Happy with child’s reading progress [PHQ -- 27]
2. Read to child [PHQ – 19]
3. Support child doing homework [PHQ – 30 and 31]
Pupils
1. Pupil attendance [PRQ – 25]
2. Pupil do homework on days not at school [PHQ – 29]
Heterogeneous Impacts
At the pupil level, we could expect two opposing heterogeneous treatment effects of the EGRS and RSP interventions, based on baseline pupil reading proficiency. The scripted lesson plans require streaming by ability within the same classroom and provides opportunity for individualized attention and could benefit children who have otherwise been left behind. However, at the same time the scripted lesson plans are aligned to the national curriculum, which prescribes an ambitious pace in the South African context. The worst-performing pupils might benefit less if the teachers who follow the scripted lesson plans now progress at too fast a pace. Furthermore, boys/girls might benefit more/less from the individualized attention. Finally, the emphasis on individualized attention and tracking means that pupils might benefit more from the scripted lesson plans when the class size is large.
For the RSP evaluation, we expect that the success of the interventions depend on teacher motivation, prior levels of effort, and ability. The scripted lesson plans will only be applied by teachers who have a sufficient level of intrinsic motivation. Related, teachers who have a higher burden of lesson preparation are most likely to switch to scripted lesson plans, because they have most to gain from scripted lesson. Furthermore, teachers may need a sufficient baseline level of reading proficiency in order to effectively apply the scripted lesson plans. On the other hand, the scripted lesson plans might be too restrictive for exceptional teachers who are effective at adjusting their instruction to the needs of the classroom and ability of the pupils.
We will therefore examine the following possible interactions:
Learner level
1. Learner gender [learner information form]
2. Learner age [learner information form]
3. Learner attendance of grade R [PHQ 15]
4. Learner attendance of other ECD [PHQ 18]
5. Learner baseline performance [pupil baseline test]
6. Learner attendance [PHQ 25, LWQ 2]
7. Learner speaks English at home [LWQ B1
8. Learning when not at school [PHQ 29 ,30; LWQ 2.2]
9. Learner reported amenities and possessions [LAL 1]
10. Learner reported books in home [LAL 2]

Teacher level
1. Teacher characteristics:
a. Education level [TRQ 2.5]
b. Training in Foundation Phase [TRQ 2.8]
c. Years of experience [TRQ 2.3]
2. Teacher Attendance [TRQ 4.1]
3. The extent to which teachers feel supported and recognized in their work [TRQ 5.2l]
4. The sufficiency of the classroom LTSM [TRQ 3.1, 3.2, 3.3., 3.4]

School-level
1. Socio-economic background of school [administrative data]
2. School resources [SFO Section 2]
3. School urban / rural [administrative data]
4. School district [administrative data]
5. Pupil-teacher ratio [administrative data, SPQ section 3]
6. School days lost [SFO 5.1, SPQ 7.5, 7.7, 7.8, 7.11)]
7. Teacher absenteeism [SPQ 5.1]

Parent level
1. Identity of the parent/guardian: [PHQ 8]
E.g., single mother/grandmother; child-headed household.
2. Education level [PHQ 11,12]
3. Parent employment [PHQ 13]
4. Household amenities and possessions [PHQ14]
5. Learner use of English at home [PHQ 21, 22]

EVALUATION DESIGN EGRS I
The EGRS I was designed as a randomized control trial (RCT) with three treatment arms and one control arm. A total of 230 schools were randomly assigned to one of the intervention groups (50 schools each), or the control group (80 schools). To evaluate the effectiveness of the interventions, a random sample of 20 learners were selected in Grade 1 in 2015 and were tracked into Grade 2. Learners’ reading proficiency was tested at the start of Grade 1, at the end of Grade 1, and again at the end of Grade 2.
Treatment Assignment and Random Selection for EGRS I impact evaluation
The sampling process for the EGRS I began with 458 primary schools registered in 2014 administrative data in the districts of Dr. Kenneth Kaunda and Ngaka Modiri Molema. The sampling started by excluding relatively affluent schools (those in quintiles 4 and 5). Next, schools were excluded in which the language of instruction in the Foundation Phase was not Setswana. Schools were also excluded if they were missing in the 2014 Annual National Assessment (ANA) dataset. An additional eight schools were excluded because they took part in the EGRS pilot. Small schools (fewer than 20 Grade 1 enrolments) were also excluded because many of them practice multi-grade teaching rendering the scripted lesson plans less appropriate. Larger schools (more than 180 Grade 1 enrolments) were also excluded to limit intervention costs. Three more schools were excluded after the North West PED checked the list of schools and found specific problems with these schools (e.g., the school had been closed down, or a particular conflict around school management was occurring in a school). After all of these exclusions 235 eligible schools remained. Using a random number generator, five schools were selected as possible replacement schools, with the remaining 230 schools constituting the main sample.
To increase power and assure balance between treatment arms, stratified randomization was performed. Ten strata of 23 similar schools each were created, based on school size, socio-economic status, and previous performance in the Annual National Assessments. Within each stratum, five schools were randomly assigned to each treatment group and eight to the control group. In total, 50 schools were assigned to each treatment group and 80 to the control group. Sample size calculations indicated that this sample is sufficient to identify a minimum effect size of 0.21 standard deviations when comparing a treatment group with the control group and a minimum effect size of 0.23 standard deviations when comparing two treatment groups. These calculations are based on data collection from 20 Grade 1 learners and assume a 95% confidence interval, an alpha value of 0.8, an intra-class correlation coefficient (rho) of 0.3 and a correlation between pre- and post-test scores of 0.7.
EVALUATION DESIGN RSP
A sample of 214 schools that were part of the EGRS and part of the RSP 2018 baseline data collection is included in the impact evaluation. This includes all intervention groups. Lastly, 15 schools that were part of the original EGRS sample were not included in the RSP baseline because they had converted to multi-grade schools.
Schools were assigned to RSP treatment groups as follows. The 50 schools that were in the parent group under EGRS were assigned to be the comparison group for RSP, because the parent intervention had no impact. The comparison group received training and LTSM only. In addition, 80 schools that were not part of EGRS received the same (comparison) interventions under RSP. This group is not part of the RSP evaluation design, as they were not included in the baseline evaluation and have not received an intervention to be evaluated. Schools in the three remaining EGRS groups - Training (50 schools), Coaching (50 schools) and Control (80 schools) - received training and coaching under RSP. In addition, half of the schools in each group were allocated randomly to receive a School Management Training (SMT), in addition to coaching. Without schools dropping out, this procedure should have resulted in 90 schools receiving coaching alone, 90 schools receiving coaching and SMT and 50 schools receiving only training and LTSM, received by all other groups. However, approximately 60 schools dropped out of the study, according to the most recent (January 2021) DBE database. Table 2 shows the number of schools remaining in the RSP and EGRS groups.
See Tables 2 and 3 in the analysis plan for a provisional breakdown of the Number of schools in EGRS and RSP treatment groups

Schools were randomly assigned using a random number generator in statistical software (STATA).

Randomization was conducted at the level of school, and at the level of the learner

Yes

EGRS Grade 6 learners in 229 schools
RSP Grade 3 learners in 214 schools
RSP Grade 2 learners in 214 schools

12 EGRS Grade 6 learners in 229 schools = 3664 observations 16 RSP Grade 3 learners in 214 schools = 3424 observations 16 RSP Grade 2 learners in 214 schools = 3424 observations Parents of learners assessed in 229 schools = 10283 2 Grade 3 teachers in each of the 229 schools = 458 observations 2 Grade 7 teachers in each of the 229 schools = 458 observations 1 Principal in each of the schools = 229 observations

EGRS Treatment 1 schools (Training, scripted lessons, graded readers) = 80
EGRS Treatment 2 schools (Reading coaches, scripted lessons, graded readers) = 50
EGRS Treatment 3 schools (parent involvement) = 50
EGRS Control schools = 50
RSP Treatment 1 schools (Training, learning and teaching materials) = 43
RSP Treatment 2 schools (Training, learning and teaching materials, coaching) = 140
RSP Treatment 3 schools (Training, learning and teaching materials, coaching, school management training i.e., a subset of Treatment 2 schools) = 70

SAMPLE SIZE CALCULATION Sample size calculations are conducted to inform design decisions. Below we calculate the number of learners required to achieve a reasonable minimum detectable effect size (MDES) of approximately 0.21 SDs (the MDES for the originally-designed EGRS comparison between treatment and control). For the RSP Grade 3 and EGRS Wave 6 evaluations, the sample size is determined by the number of learners who were assessed at baseline and who are present on the day of assessment. We based the calculations below on the sample sizes estimated in and using a longitudinal design (i.e., controlling for baseline values). The RSP Grade 2 evaluation uses a cross-sectional design (i.e., there is no baseline) and therefore relies only on a comparison across groups being made at a single time point. We have also conducted sample size calculations (and associated MDES) for two evaluation approaches for both EGRS and RSP. In both cases, we estimate the MDES for the original, longitudinal design and also include an estimate of the MDES for a cross-sectional design that includes a ‘top-up’ of students. It is clear from the results in Table 5 that the marginal benefit of the cross-sectional top-up is minimal and we therefore recommend using the longitudinal design. Our recommended samples sizes are therefore ~12 students in EGRS Grade 6; ~16 students in Grade 2 and ~16 students in Grade 3 for RSP. Additional assumptions and explanations are provided in the section below Table 5.   Table 5 Sample size calculations (longitudinal versus cross-sectional) Assumption EGRS Longitudinal EGRS Cross-Sectional RSP Grade 3 Longitudinal RSP Grade 3 Cross-Sectional RSP Grade 2 Cross-Sectional Zα = 1.6450 1.6450 1.6450 1.6450 1.6450 Zβ = 0.8416 0.8416 0.8416 0.8416 0.8416 Group 1 Total Schools 80 80 70 70 70 Group 2 Total Schools 50 50 43 43 43 Group 1 Students/School 12 20 16 20 16 Group 2 Students/School 12 20 16 20 16 Group 1 Students 960 1600 1120 1400 1120 Group 2 Students 600 1000 688 860 688 ICC 0.2 0.2 0.2 0.2 0.2 Design Effect 3.2 4.8 4 4.8 4 Corr 0.7 n/a 0.7 n/a n/a Minimum Detectable Effect Size (MDES) 0.18 0.22 0.19 0.237 0.241 RSP Impact Grade 2 (2021) Based on the equation and assumptions below, we recommend sampling 16 learners per school for a cross-sectional analysis in order to detect an effect size of 0.24 or larger. The estimate is based on the following parameters: μ_1 is the control’s mean of the Grade 2 learners’ scores. μ_2 is the treatment’s mean of the Grade 2 learners’ scores. A difference of 5 cwpm (μ_1-μ_2=5) is equivalent to an effect size of 0.2, with a standard deviation of 25. σ_1 is the standard deviation of the control’s scores. 25 σ_2 is the standard deviation of the treatment scores. 25 1-β is the power set at 0.8 α is the alpha 0.05 ρ is the intraclass correlation. 0.2 m is the average cluster size (average number of learners sampled within each school). [to be solved for] RSP Impact Grade 1 (2018) – Grade 3 (Grade 4 or lower in 2021) Given the current sample size of schools and assuming an attrition rate of 17 percent, we would expect an average of 16 learners who were assessed in Grade 1 in 2018 to be assessed in 2021. Given the following assumptions below, we expect the minimal detectable effect size approximately 0.18 with a longitudinal analysis. This translates to an approximate gain of five correct words per minute in the measure of reading fluency of the intervention groups over the comparison group. μ_1 is the control’s mean gain in learners’ scores from Grade 1 to Grade 3. μ_2 is the treatment’s mean gain in learners’ scores from Grade 1 to Grade 3. Assuming a difference of 5 cwpm, (μ_1-μ_2=5) σ_1 is the standard deviation of the difference in comparison group scores. 22 σ_2 is the standard deviation of the difference in treatment group scores. 22 1-β is the power 0.8 α is the alpha 0.05 ρ is the intraclass correlation. 0.2 m is the average cluster size (average number of learners sampled within each school). 15 r is the correlation coefficients between the baseline (2018) and endline (2021) = 0.7 EGRS 1 Impact Grade 1 (2015) – Grade 4 (2018) – Grade 6 (Grade 7 or lower in 2021) The current sample size of schools and assuming an attrition rate of 40 percent, we would expect on average 12 learners (who were assessed in Grade 1 in 2015) per school will be assessed in 2021. Given the following assumptions below, we expect the minimal detectable effect size would be approximately 0.18 for this longitudinal analysis. μ_1 is the control’s mean gain in learners’ scores from Grade 1 to Grade 6. μ_2 is the treatment’s mean gain in learners’ scores from Grade 1 to Grade 6. μ_1-μ_2 to be solved for. A difference of 8 cwpm with a standard deviation of 32 would give an effect size of 0.25. σ_1 is the standard deviation of the difference in control’s scores. 32 σ_2 is the standard deviation of the difference in treatment scores. 32 1-β is the power 0.8 α is the alpha 0.05 ρ is the intraclass correlation. 0.2 m is the average cluster size (average number of learners sampled within each school). 12 r is the correlation coefficients between the baseline (2015) and endline (2021). 0.7 (This applies to the longitudinal design only)

EGRS+RSP+Pre-Analysis+Plan+2021.pdf