Improving the teaching of English as a first additional language in South Africa.

Both interventions aim to apply the same set of instructional practices in the teaching of English as a First Additional Language (EFAL) in the grade 1, 2 and 3 classrooms. Both interventions provide teachers with clear, scripted lesson plans, which are aligned to the official South African government curriculum. The lesson plans incorporate the use of learning support materials, including the government provided workbooks as well as certain additional materials (graded reading booklets, flash cards, big books, posters, etc.), provided by the program. Graded reading booklets were introduced in the second year of the study (as per the recommendation of the CAPS) as a key resource for the teacher to use in group-guided reading and individual work, so as to facilitate reading practice at an appropriate pace and sequence of progression.

Intervention 1 initially introduces the teachers on how to use the lesson plans and accompanying materials through a two-day central training session at the start of the year. During the year, cluster training sessions are held for one-day prior to the start of each academic term to additionally train the teachers on the use of certain core methodologies in the lesson plans. In addition to the training sessions, ongoing support is provided to teachers through ‘specialist’ reading coaches. The on-going support takes on the form of on-site visits by the reading coach on a monthly basis to assist teachers with the implementation of the lesson plans.

Exactly the same set of instructional materials (graded reading booklets, flash cards, big books, posters, etc.) is provided to Intervention 2 schools. However, instead of providing the scripted lesson plans in a paper-based format, the scripted lesson plans are provided in an electronic format on tablets to teachers. In addition to the scripted lesson plans, the tablets also contain short video tutorials for the teachers on teaching practice, audio clips on the pronunciation of the phonics sounds, audio clips of the songs and rhymes and examples of learners’ work. Furthermore, instead of a reading coach visiting the schools on a regular basis, the intervention has a virtual coach who provides on-going support to teachers through phone calls, Whatsapp and SMS messaging. The teachers in intervention 2 are initially introduced to the lesson plans and materials through a three-day central training session at the start of the year, and also receive one-day cluster training sessions at the start of each academic term.

2017-03-01

2019-12-01

1. English language proficiency.
2. English reading proficiency.

Both indices are aggregated using principal component analysis, and then standardized to have a control mean of zero and standard deviation of one. The indicators that constitute each index are based on oral and written assessments of students and include:
1. Language proficiency: (i) English expressive vocabulary test; and (ii) English listening comprehension test
2. Reading proficiency: (i) Oral Reading Fluency, (ii) Oral Reading Comprehension; (iii) Written comprehension.

1. Home language proficiency
2. Performance in a mathematical test.
3. Implementation fidelity
4. School support
5. Teacher behavior.
6. Skill acquisition.
3. Teachers' use of resources provided by the program
4. Total time (hours per week) dedicated to teaching English as a First Additional Language
5. Total time (hours per week) dedicated to teaching home language.
6. Teachers' pedagogical knowledge.
7. Teachers' perceived difficulty of teaching the different techniques.
8. Teachers' enjoyment of teaching English as a First Additional Language.
9. Number of written exercises completed by students in the year.
10. An index of the English print-richness of the class environment.

In order to test whether the treatments had any crowding-out or spillover effects on the other subject areas, we will also look at the following outcomes:
1. Home language reading proficiency, constructed using principal component analysis with the following indicators:
a. Letter recognition
b. Oral Reading Fluency
c. Oral reading comprehension.
d. Written comprehension.
2. Performance in a simple mathematics test.

In order to test for mechanisms based on our theory of change, we will also examine the impact on a series of intermediate outcomes:

3. Implementation fidelity
a. Received training [[TRQ 6.4]
b. Frequency that a coach/mentor (i) observes a teacher in EFAL [TRQ 6.6.4], (ii) demonstrate teaching practice [TRQ 6.7.4], or (iii) provide a compliment or praise [TRQ 6.8.4]
c. Have access to graded readers in the classroom [TRQ 4.2.1]
d. Total time spent teaching:
i. English as a First Additional Language [TRQ 3.4]
ii. Home Language [TRQ 3.5]
4. School support.
a. Frequency that the school principal, head of department, or subject advisor (i) observes you teach EFAL [TRQ 6.6.1 - 6.6.3], (ii) demonstrate teaching practice [TRQ 6.7.1 – 6.7.3], or (iii) provide a compliment or praise [TRQ 6.7.1 – 6.7.3]
5. Teacher behavior
a. Use of resources provided by the program, including (i) proportion of teachers who use the Graded Readers daily [TRQ 4.2.3], (ii) proportion of teachers who use Lesson Plans provided by an NGO [TRQ 4.1.3], and (iii) print richness of the environment [TRQ 8]
b. The proportion of the teaching practices that are conducted at the correct weekly frequency [TRQ 4.7; 4.10].
c. The completion of more writing exercises [TRQ 9.3]
6. Skill acquisition: The proportion of teaching activities that a teacher finds difficult to implement [TRQ 4.8]

Through a process of elimination we developed a sampling frame of 180 eligible schools. Beginning with 731 primary schools registered in the 2016 administrative data in the districts of Ehlanzeni and Gert Sibande we firstly excluded relatively affluent schools (those in quintiles 4 and 5). Next, we excluded schools in which the language of instruction in the Foundation Phase was neither Siswati, nor isiZulu. We also excluded schools which were missing in the 2014 Annual National Assessment (ANA) dataset. We further excluded particularly small schools (fewer than 30 grade 1 enrolments) since many of these schools would practice multi-grade teaching rendering the grade-specific lesson plans less appropriate. We also excluded particularly large schools (more than 160 grade 1 enrolments, or more than 3 classes in grade 1, or classes with more than 60 learners in) to limit intervention costs. After all of these exclusions 193 eligible schools remained. Using a random number generator, we then excluded 3 further schools to remain with a sample of 190 schools. The 190 school sample included a sample of 10 replacement schools (1 in each of the strata) should the need arise to drop one of the sample schools. Thus we obtained the sampling frame of 190 schools.

To increase power and assure balance between treatment arms, we performed stratified randomization. We created 10 strata of 19 similar schools based on school size, socio-economic status, and previous performance in the Annual National Assessments. Within each stratum, we then randomly assigned 5 schools to each treatment group, 8 to the control group and 1 as a replacement school. Thus we randomly assigned 50 schools to each treatment and 80 to the control.

Randomization done in office on Stata.

School level.

Yes

180 schools.

At baseline we have a sample of 3600 students: 20 students per school.

80 schools control, 50 schools each of the two treatment arms.

Given that we collect data on 20 grade 1 learners per school, this sample should be sufficient to identify a minimum effect size of 0.21 standard deviations when comparing a treatment group with the control group and a minimum effect size of 0.23 standard deviations when comparing two treatment groups. These calculations assume a 95% confidence interval, an alpha value of 0.8, an intra-class correlation coefficient (rho) of 0.3 and a correlation between pre- and post-test scores of 0.7.

The Second Early Grade Reading Study in South Africa: Pre-Analysis Plan