Experimental Design
1. Job advertisements for after school tutors were posted throughout the schools and villages in the catchment area beginning from January 25th, 2015. The NGO advertised vacancies through posters which describe the job and work conditions. High school graduation was the minimum requirement for the position, and applicants were asked to submit an application form along with their Malawi School Leaving Exam transcript (secondary school diploma). Applications were received at 31 primary schools in the catchment area and AFF office. AFF invited applicants to participate in the recruitment meeting in which they participated in the baseline survey, took a math test, and performed a teaching demonstration.
2. The math test consists of 90 questions, which are chosen from 4th to 12th grade textbooks. For the teaching demonstration, applicants were given 2 math problems from the secondary school level (9-10th grade level). Each applicant had 10 minutes to prepare the demonstration and 10 minutes to present it. Several pairs of trained AFF staff members were randomly assigned to the applicants in order to evaluate each applicant’s teaching skill. Specifically, they evaluated overall performance based on the applicant’s engagement with the audience, delivery of mathematical elements in the problems, time management, and enthusiasm.
3. AFF randomly divides applicants into two groups. 173 were assigned to the “math test” group, while the other 174 were assigned to the “teaching demonstration” group. 132 out of 173 in the math test group were hired based on their math test score as after school teachers. 130 out of 174 in the teaching demonstration group were hired based on their teaching demonstration evaluation.
4. Selected after school teachers were required to complete a one-week-long training, during which they were subjectively evaluated by evaluators (regular employees of AFF). Trainees participated in a mock teaching session, where they presented how to solve a pre-assigned math problem from an assigned math course. Evaluators then scored trainees on a scale from 1 to 3 in terms of engagement with the audience, delivery of mathematical elements in the problems, time management, and enthusiasm.
5. The first semester of the experiment is the third semester of the academic year 2015/16 (April 11, 2015 - July 15, 2016). 122 grades (4th-7th grade) in 31 schools were randomized to either tutoring or no tutoring based on students’ grade and average class level computed from students’ math exam score.
Group 1 (42 grades): no tutoring (5,303 students)
Group 2 (78 grades): tutoring (6,396 students). Out of 6,396 students, 2,849 students were randomly assigned and 3,547 students were not assigned to tutoring.
6. As a result of the above process, 2,849 students out of 6,396 students were randomly assigned to after school tutoring in the third semester of the academic year 2015/2016. The remaining students will be eligible for the tutoring in the following semesters.
7. Each teacher was also randomly assigned to 2 out of 414 after school classes. Each class was held twice a week, either Monday-Wednesday or Tuesday-Thursday. The duration of the tutoring program was 7 weeks per semester, and eligible students took 13 classes in total.
8. There are four tutoring class sizes: 4, 6, 8, and 10. These four class sizes were randomly distributed within each school-grade. For example, when one school-grade consists of 35 students in the tutoring treatment, each of the four class sizes (4, 6, 8, and 10) is initially assigned in a random order.
9. Students were then assigned to the classes in descending order of their baseline math scores. As a result, they were in the same class as those with a similar baseline math score.
10. The collaborating NGO commits to hire the after school teachers for at least three semesters. In the second semester of the experiment (i.e. the 1st semester of the academic year 2016/17), 120 grades (5th-8th grade; students moved on to the next grade as the school year changed) in 31 schools were randomized to either tutoring or no tutoring. School-grades which were not in the tutoring group (42 grades) in the first semester of tutoring were automatically included in the tutoring group. Of the school-grades which received tutoring in the first semester (78 grades), 50% (38 grades) were randomly selected for the tutoring group, with the remaining 40 grades assigned to the no tutoring group. The random assignment of tutors, class sizes, and students was conducted in the same manner as that of the first semester. We will continue to provide the tutoring service to students through the third semester of the experiment (the 2nd semester of the academic year 2016/17).