Comparing the Effectiveness of Two Teacher Training Programs: Experimental Evidence from Tanzania

Both interventions are in-service teacher training programs implemented by the Swiss NGO Helvetas, that target primary school teachers teaching math to 6th and 7th graders (among others).

The first intervention, which is labelled as "SITT" (School Based In-Service Teacher Training), builds around a 5-days workshop with local education experts, that aim to introduce new teaching techniques (pedagogy) but also repeat selected concepts from the official primary school curriculum. Moreover, all participants obtain specifically developed manuals covering Algebra (>100 pages), Numbers (>200 pages), Geometry (>100 pages), and Fractions (>100 pages). These manuals serve as basis for the teacher training courses and can be used by teachers as a guide and source of inspiration throughout the school year; the focus of these manuals lies on interactive and practice-oriented learning. SITT further aims at exploiting multiplier/cascade effects: In particular, participants are asked to form peer-learning groups at their school and introduce the new techniques to their fellow teachers. To ensure compliance, school authorities (incl. head teachers, quality assurance officers and district education officers) agreed to closely monitor the functioning of these peer-learning groups at their school(s); this close monitoring should catalyze trickle down effects of the intervention to all math teachers that are not directly treated.

The second intervention arm, which is labelled as "SITT-D", basically mirrors the original SITT-program, but adds a “Digital” component. Additionally to the core elements of SITT, the 65 participants of SITT-D will receive a tablet with computer-assisted learning software. They will be introduced to the software during the initial 5-days workshop, and receive assignments to refresh those concepts that were answered worst in the baseline assessment. In particular, the following elements to the original SITT-version are added: (i) An introduction to the software during the initial 5-days workshop, (ii) computer-assisted learning assignments as part of their action plans, (iii) two additional two-days meetings in 2020 (~April & ~September) to discuss the assignments and (iv) regular submission of automated outputs produced by the CAL-software to monitor their work with the software. Both the self-studying and the attendance in workshops are (weakly) incentivized, meaning that the participants will get awarded the IT-hardware, in case their participation rate crosses a certain threshold.

2020-02-24

2020-10-31

Math skills of primary school teachers measured via pencil and paper assessments covering the local curriculum of grades 2 to 7. We also assess their students' math ability at the end of the school years 2021 and 2022 based on national standardized math exams in grade 7 conducted by the National Examimations Council of Tanzania (NECTA).

Classroom observation will be conducted where we collect data on teacher attendance and pedagogical aspects during the lessons. These observations are based on the "Teach" observation tool of the World Bank and adapted to the context.

Starting point are 318 primary schools in four Tanzanian districts, i.e. Mbulu DC (Manyara), Mbulu TC (Manyara), Karatu (Arusha) and Siha (Kilimanjaro).

We exclude 49 of these 318 schools for one or several of the following reasons: (i) the school is run by a private entity, or (ii) the school did not participate in the Standard Four National Examination 2018, or (iii) less than 10 students participated in the Standard Four National Examination, or (iv) the school has less than 6 teachers.

After this first pre-slection, 269 primary schools remain in our sample. We sort these 269 schools along an accessibility index incorporating an ordinal assessment by the District Education Offices and Google Map driving times to the district center. Based on this ranking, we exclude the 49 least accessible schools.

Following this preselection, 220 schools remain in our sample. For these 220 schools the District Education Officers provided us with name lists of all teachers that (regularly) teach math to 4th, 5th, 6th, or 7th graders. The District Education Officers, in consultation with the schools, designated one of the math teachers as "Program Teacher". While we did not communicate any specific selection criteria for these "Program Teachers" the purpose of doing so was well explained: Only one teacher per school, that is the "Program Teacher" will potentially be assigned to Treatment 1 (i.e. SITT), or Treatment 2 (i.e. SITT-D), or the Control Group. Moreover, we asked the involved authorities and the selected teacher to sign a "Declaration of Intent", specifying that we expect these "Program Teachers" (i) to remain in the same school until the end of school year 2021, (ii) and that they will teach the same class of 6th graders (as of 2020) until they reach the Primary School Leaving Examination in 2021.

Moreover, we use the list of math teachers provided by the District Education Officer to sample one additional teacher per school as participant in our math assessments. We aim to use the data obtained from these teachers for two purposes: First, to gain insights on the math content knowledge for a representative sample of math teachers (standard 4--standard 7) at the pre-selected schools. Second, to be able to measure peer-to-peer spillovers/multiplier effects at the SITT- and SITT-D-treatment schools.

This procedure yields a list of 440 math teachers from 220 different schools. 220 of these teachers, i.e. one per school, was designated as "Program Teacher"; these 220 teachers are randomly assigned to Treatment 1 (SITT, 65 "Program Teachers"), Treatment 2 (SITT-D, 65 "Program Teachers"), or the Control Group (90 "Program Teachers"). The random assignment was implemented with STATA, and we stratify on average test scores of students in the National Standardized Math Assessment in grade 7, teacher baseline test scores, region.

To assess the impact of the two interventions on the teachers' math content knowledge, a baseline assessment with the 440 teachers will be conducted in November 2019 and an endline assessment in November 2020. The scores of the "Program Teachers" will inform us on the direct impact of the interventions on the teachers' content knowledge, while the scores of the additional (regular) teachers allows us to measure potential spillovers.

To assess the impact of the interventions on the students' math abilities, we will rely on the standardized national assessments conducted with 4th-graders (NSFA, "baseline") and 7th-graders (PSLE, "endline") administered by the The National Examinations Council of Tanzania (NECTA).

Randomization is done in the office using Stata.

We randomize on the school/teacher level stratifying on average test scores of students in the National Standardized Math Assessment in grade 7 (grouped in terciles), teacher baseline test scores (two groups) and on districts (Mbulu town, Mbulu district, Karatu and Siha). Note that each school nominates only one teacher for the program. Hence, for teachers the randomization of the treatments was not clustered, while for students it is clustered on the teacher level.

No

220 schools.

220 designated "Program Teachers", who are nominated by the Education Authorities to participate in the program; these are randomly assigned to Treatment 1 (SITT, 65), Treatment 2 (SITT-D, 65) or the Control Group (90).

220 randomly sampled math teachers, who regularly teach standard 4, standard 5, standard 6, or standard 7 classes. These allow us to measure within school spillovers.

220 "Program Teachers", 220 additional teachers to measure spillovers. The math abilities of their students are also assessed, as we plan to exploit the Primary School Leaving Examinations conducted by NECTA in 2021 and 2022. While class sizes vary considerably and change at the start of new school years, we expect on average 40-50 students per teacher, i.e. about 8'800-11'000 students per wave.

65 teachers are assigned to Treatment 1 (SITT), 65 teachers are assigned to Treatment 2 (SITT-D) and 90 teachers are assigned to the control group.

As we expect on average 40-50 students per teacher, we estimate to evaluate the direct effect on students' math abilities based on the results of 8'800-11'000 Primary School Leaving Exams conducted by NECTA in 2020.

MDE_TEACHERS=0.2 -- 0.32 standard deviations. Calculations based on formula by Bloom (2007) and following parameter values: power=80%; alpha (level of significance)=0.05; R-squared: 0.5--0.8 (based on empirical values in Brunetti et al. 2019, see RCT ID: AEARCTR-0004092); P (share of control units)=0.58; n (total observations)=155. MDE_STUDENTS=0.14 -- 0.19 standard deviations. Calculations based on formula by Bloom (2007) and following parameter values: power=80%; alpha (level of significance)=0.05; R-squared (between): 0.13--0.56; R-squared (within): 0.16--0.47; Rho (Interclass correlation)=0.19--0.20; P (share of control units)=0.58; J (total clusters)=155; n (observations per cluster)=40.