Engineering and Evaluating Cascading Effects in Teacher Training Programs: Experimental Evidence from Tanzania

The intervention is an in-service teacher training program, implemented by the Swiss NGO Helvetas in collaboration with the Tanzania Teachers' Union (TTU). The SITT program (School Based In-Service Teacher Training) targets primary school teachers teaching math and science to 6th and 7th graders (among others) as well as head teachers.

The intervention builds around a 5 days workshop with local education experts. The workshop aims to introduce new interactive teaching techniques to math and science teachers. Furthermore, selected concepts from the official primary school curriculum are repeated. Moreover, all participants obtain specifically developed manuals, focusing on interactive and practice-oriented learning. The SITT program further aims at exploiting cascading effects: In particular, participants are asked to form peer-learning groups at their school and introduce the new techniques to their fellow teachers. Preliminary results from Brunetti et al. 2020 (see AEARCTR-0004959) suggest that the program has not yet managed to produce meaningful cascading effects. Therefore, the following changes have been introduced to the program:

(i) In previous implementations of the program, only one math teacher per school participated in the program. As preliminary results suggest that a single math teacher was not able to measurably cascade his or her new knowledge to their peers at their local school, the intervention will now target two teachers per school (one math and one science teacher). A lesson will be dedicated to exchange on how the two teachers can work together as a team to share their knowledge with peers, thus becoming agents of change. This might be a promising approach, as many studies find teachers working as a team to have positive effects on all involved stakeholders (see e.g. Baeten 2014; Rytivaara 2019).

(ii) Up to now, head teachers have had no role in the SITT program. However, the leadership by head teachers is often viewed as a crucial component for educational success (Böhlmark et al. 2015). They provide encouragement, resources, and support for their teaching staff, organize schedules, and represent their school vis à vis relevant stakeholder. Numerous recent studies show that capable head teachers can have a substantial positive impact on school performances and pupils learning outcomes (see e.g., Bloom et al. 2015; Böhlmark et al. 2016; Di Liberto 2015). To better account for the crucial role of head teachers, the implementer will invite the head teachers of program schools to join the SITT workshops. The aim is to familiarize head teachers with the program and foster a conceptual understanding of the new teaching methods, raise their awareness about the benefits of these methods, and provide guidance on how head teachers can support cascading at their local school. To achieve this, head teachers will participate in the main workshop (to familiarize them with the program) and a parallel session focused on their role as supporters of the cascading approach.

(iii) The results of the previous evaluation of the program by Brunetti et al. 2020 (see AEARCTR-0004959) will be summarized on a single page document. This document can be distributed to participants to motivate them by informing on the benefits of the program. Furthermore, participants can use the document as a tool to communicate with their peer teachers when cascading the program at their local school.

The 5-day workshop will be followed up by semi-annual one- or two-day refresher meetings to provide participants (head teachers and math / science teachers) a platform to exchange on ideas, ask questions, and repeat selected instructional techniques with SITT instructions.

2023-02-06

2024-07-31

Pupils math and science skills at the end of the school years 2023 and 2024 based on national standardized math and science exams in grade 4 and grade 7, administered by the National Examinations Council of Tanzania (NECTA).

Classroom observations will be conducted where we collect data on teacher attendance and pedagogical aspects during the lessons. These observations are based on the "Teach" observation tool of the World Bank and adapted to the context.

Starting point are all 369 primary schools in five Tanzanian districts. We exclude 17 of these 369 schools because they are run by private entities. After this first pre-selection there are 352 schools remaining: 84 schools in Ruangwa District Council (Lindi region), 113 schools in Nachingwea District Council (Lindi region), 31 schools in Mtwara Mikindani Municipal Council (Mtwara region), 46 schools in Newala Town Council (Mtwara region) and 78 schools in Songea District Council (Ruvuma region).

Within each district we sort the schools along the distance from the school to the district headquarters. Based on this ranking we select the 40 closest schools. As there are only 31 schools in Mtwara Mikindani Municipal Council (all within close proximity to the district headquarters), we select all 31 schools. As Nachingwea District Council has the most schools, 49 schools from this district are selected. This procedures leaves us with 200 schools in our sample.

For these 200 schools the head teacher is nominated for participation in the SITT program. Furthermore, in consultation with the District Education Officer, each school could nominate two regular teachers (one math and one science teacher) for participation in the program. When nominating the two regular teachers, schools are asked to adhere to the following selection criteria: (C1) Both nominees must continue teaching at their current school until the end of 2024. (C2) Both nominees should teach pupils of grade 6 in 2023 and grade 7 in 2024. (C3) Both nominees should not teach pupils of grade 4 and below in 2023 and 2024. All nominees (head teacher and the two regular teachers) are invited to participate in a sensitization event scheduled in early November 2022.

During the sensitization events in early November 2022, the nominated head teachers (200), the nominated math teachers (200) and the nominated science teachers (200) are introduced to the SITT program. The introduction covers the following points: (i) general presentation about the SITT program, (ii) information on the scope and the goals of the evaluation study. (iii) Furthermore, participants will be informed about the criteria for participating in the study (see (C1) to (C2) of previous paragraph) and about their option to opt out of the study at any time. The presentation emphasizes that all participants must adhere to these criteria, no matter whether their school is assigned to the control or treatment group. After the presentations, the teachers will be asked to give written consent to participate in the program and its evaluation. Teachers who provide written consent to participate in the program and its evaluation will be asked to fill in a socio-demographic survey. Teachers who could not participate in the sensitization event are contacted via an online form, where they are asked to provide written consent to participate in the study and fill in the socio-demographic survey.

The 200 participating schools are then randomly assigned to either treatment group (participating in the SITT program, 100 schools) or control group (not participating in the SITT program, 100 schools). If a teacher did not give consent to participate in the study (or cannot be reached by online form), he or she will not be part of the evaluation study. The random assignment is implemented with STATA, we stratify on average test scores of students in math and science in the Standard Four National Assessment (SFNA) and on district.

As schools are randomly grouped into treatment and control group, any aggregate differences in the post-intervention outcomes can be attributed to the intervention. The analysis will distinguish between a direct effect and an indirect effect (i.e. an effect through cascading) of the intervention. To assess the direct effect of the intervention we will rely on two outcomes: First, classroom observations will be conducted in 2023 and 2024 to analyse whether observed teaching practices systematically differ between teachers who participated in the intervention and those who did not. Therefore, classroom observations will be scheduled for classes taught by (i) teachers who directly participated in the SITT program, (ii) teachers from control schools who did not participate in the program at all (neither directly nor indirectly through cascading). Second, an analysis of National Examination Results administered by the National Examinations Council of Tanzania (NECTA) will yield insights on whether the intervention improved learning outcomes in science and mathematics among pupils. To estimate the direct impact of the intervention we will rely on the annual Primary School Leaving Examinations (PSLE) conducted with seventh graders. The participating teachers (treatment and control schools) will have agreed to teach pupils of grade 6 in 2023 and pupils of grade 7 in 2024. Therefore, pupils who will take the PSLE exam by the end of 2024 will have been taught by the same teachers for two consecutive years. From a design perspective, participation criteria (C1) and (C2) are therefore pivotal to estimate the direct effect of the intervention. By estimating the direct effect of the intervention, we will partially replicate Brunetti al. 2020 (see AEARCTR-0004959). Important: It will not be possible to directly match teachers to pupils. This means we will not know which share of 7th graders will have been taught by teachers who are part of the evaluation. However, most schools in the sample will only have a single class per grade (the median school in the sample has 53 pupils per grade which corresponds to average class size).

The main contribution of this project is to produce insights on how to exploit cascading effects in teacher training programs. Preliminary results (Brunetti et al. 2020) suggest that previous implementations of the intervention did not manage to produce measurable cascading effects. The main focus of this project is therefore to experimentally evaluate the deliberate design changes (see paragraph on "Intervention") to improve on this. To that end, we will asses whether two teacher training participants (math and science teacher) with the support of their head teacher can spark measurable cascading effects that reflect in more engaging instructional routines among peer teachers. To test this, we will compare the results of classroom observations from classes taught by (i) teachers who did not participate directly but were exposed to the intervention through cascading, to (ii) teachers from control schools who did not participate in the program at all (neither directly nor indirectly through cascading). Finally, Standard Four National Assessments (SFNA) conducted with fourth graders will serve as basis to estimate the indirect impact of the intervention through cascading on pupils learning outcomes. This will allow to test whether the potential behavioral changes among peer teachers who were only exposed to the program through cascading are relevant enough to accelerate the learning progress among their pupils. The participating teachers (treatment and control schools) will have agreed to not teach pupils of grade 4 and below in 2023 and 2024. Teachers in treatment schools teaching those grades can hence only benefit from the intervention through peer learning activities organized by their colleagues that attended the training (i.e. cascading). Therefore, any measurable difference in the outcomes of pupils who take the SFNA in 2024 can only be attributable to cascading of the intervention by participating teachers to peer teachers. From a design perspective, participation criterion (C3) is therefore pivotal to estimate the indirect effect of the intervention. Important: We will use SFNA results to estimate the indirect effect of the program for two reasons: First, it will not be possible to match teachers to students. Therefore, it would not be feasible to use results from PSLE as we cannot distinguish between students who were taught directly by participants of the intervention and their peers. Second, many schools only have a single class per grade. In these schools all students taking the PSLE exam in 2024 will have been taught directly by participants of the evaluation, making it infeasible to estimate indirect effects using PSLE results.

Not available

Randomization is done in the office using Stata.

We randomize on the school level stratifying on average test score (math and science) of fourth graders in 2021 in standardized national math and science exams (the Standard Four National Assessments, SFNA) grouped in terciles, and on districts (Ruangwa District Council, Nachingwea District Council, Mtwara Mikindani Municipal Council, Newala Town Council and Songea District Council). Note that for students the treatment is clustered on the school level.

Yes

There are 200 schools who participate in the evaluation.

Each school nominates the head teacher, one math and one science teacher for participation, i.e. 200 head teachers, 200 math teachers and 200 science teachers participate in the evaluation.

Schools are then randomly assigned to a treatment (100 schools, i.e. 100 head teachers, 100 math teachers and 100 science teachers) and a control group (100 schools, i.e. 100 head teachers, 100 math teachers and 100 science teachers).

200 schools, respectively 600 teachers: 200 head teachers, 200 math and 200 science teachers. The math and science abilities of their students are assessed using standardized national exams administered by the National Examinations Council of Tanzania (NECTA). Class sizes may vary considerably and change at the start of new school years. However, we expect on average 40 - 50 students per teacher, i.e. about 8'000 - 10'000 students per wave.

100 schools (resp. 100 head teachers, 100 math and 100 science teachers) are assigned to the treatment group.
100 schools (resp. 100 head teachers, 100 math and 100 science teachers) are assigned to the control group.

As we expect on average 40-50 students per teacher, we estimate to evaluate the direct effect on pupils' math and science abilities based on the results of 8'000-10'000 7th graders who will take the national PSLE in 2024.

Similarly, we estimate to evaluate the indirect effect on pupils' math and science abilities based on the results of 8'000 - 10'000 4th graders who will take the SFNA in 2024.

Minimum detectable effect size for DIRECT EFFECT: 0.12 -- 0.16 standard deviations. Calculations based on formula by Bloom (2007) and following parameter values: power=80%; alpha (level of significance) =0.05; R-squared (between): 0.13--0.56; R-squared (within): 0.21--0.47; Rho (ICC)=0.17--0.20; P (share of control units) =0.5; J (total clusters) =200; n (observations per cluster) =40. Minimum detectable effect size for INDIRECT EFFECT: 0.13 -- 0.16 standard deviations. Calculations based on formula by Bloom (2007) and following parameter values: power=80%; alpha (level of significance) =0.05; R-squared (between): 0.13--0.56; R-squared (within): 0.05 – 0.15; Rho (ICC)=0.17--0.20; P (share of control units) =0.5; J (total clusters) =200; n (observations per cluster) =40.