xNEW UPDATE: Completed trials may now upload and register supplementary documents (e.g. null results reports, populated pre-analysis plans, or post-trial results reports) in the Post Trial section under Reports, Papers, & Other Materials.

Last registered on November 23, 2016

Trial Information

Name

Affiliation

Singapore Management University

Status

On going

Start date

2016-01-25

End date

2018-06-30

Additional Keywords

Secondary IDs

Abstract

Teacher quality is one of the key determinants of students’ academic success. We examine the impact of teacher screening criteria on academic outcomes of primary school students to better hire effective teachers. Specifically, we consider two different traits of a teacher that can potentially improve academic outcomes: 1) a teacher candidate’s math test score and 2) his/her performance on a teaching demonstration. Our study takes place in the context of the after school teacher recruitment drive in rural Malawi by our collaborating NGO, Africa Future Foundation (AFF). During the pre-employment testing session, AFF conducted the math test and teaching demonstration for 347 teacher applicants. Then, AFF randomly divided them into two groups: “math test” and “teaching demonstration” group. 132 out of 173 in the math test group were hired based on their math test score as after school teachers. 130 out of 174 in the teaching demonstration group were hired based on their teaching demonstration evaluation. Hired teachers were randomly assigned to 414 classes in 31 schools, with 122 classes comprising of 4-7th graders. A class size of 4, 6, 8, or 10 was randomly determined for each tutoring class. We compare the performance of teachers (students’ test score) selected via two different channels over one year (three semesters) to shed light on effective teacher screening.

External Link(s)

Citation

Kim, Hyuncheol and Seonghoon Kim. 2016. "Hiring Effective Math Teachers: More Knowledge vs. Better Delivery ." AEA RCT Registry. November 23. https://doi.org/10.1257/rct.1687-1.0.

Former Citation

Kim, Hyuncheol, Seonghoon Kim and Seonghoon Kim. 2016. "Hiring Effective Math Teachers: More Knowledge vs. Better Delivery ." AEA RCT Registry. November 23. http://www.socialscienceregistry.org/trials/1687/history/11968.

Sponsors & Partners

There are documents in this trial unavailable to the public. Use the button below to request access to this information.

Request InformationExperimental Details

Intervention(s)

We hire after school math teachers randomly based on either their math test score or teaching demonstration and measure the impacts on students’ test scores. Below are the details of the experiment.

Africa Future Foundation (AFF) recruited after school math teachers to teach 4th to 7th graders at the onset of the after school tutoring program in the catchment district of rural Malawi (T.A. Chimutu). (As of September, 2016, students are now 5th to 8th graders.) The total number of applicants was 347. Job applicants participated in a survey which collects information on demographics, educational background, and non-cognitive traits. They also took a math test and performed a teaching demonstration which AFF used as evaluation criteria for recruitment.

Then, AFF randomly divided the applicants into two groups: 173 were assigned to the “math test” group, while the other 174 were assigned to the “teaching demonstration” group. 132 out of 173 in the math test group were hired based on the math test score as after school teachers. 130 out of 174 in the teaching demonstration group were hired based on their teaching demonstration evaluation.

After the one-week-long training, after school teachers were randomly assigned to students from 122 grades in 31 schools. In addition, AFF randomly selected two thirds of the grade-schools (for example, 80 grade-schools in the first semester) for the after school tutoring treatment. In the selected grades, a randomly selected one half of the students is eligible for after school tutoring. In this manner, everyone will be eligible at least once during the school calendar year, which consists of three semesters. In addition, students eligible for tutoring are randomly assigned to groups of tutoring class sizes of 4, 6, 8 or 10.

Africa Future Foundation (AFF) recruited after school math teachers to teach 4th to 7th graders at the onset of the after school tutoring program in the catchment district of rural Malawi (T.A. Chimutu). (As of September, 2016, students are now 5th to 8th graders.) The total number of applicants was 347. Job applicants participated in a survey which collects information on demographics, educational background, and non-cognitive traits. They also took a math test and performed a teaching demonstration which AFF used as evaluation criteria for recruitment.

Then, AFF randomly divided the applicants into two groups: 173 were assigned to the “math test” group, while the other 174 were assigned to the “teaching demonstration” group. 132 out of 173 in the math test group were hired based on the math test score as after school teachers. 130 out of 174 in the teaching demonstration group were hired based on their teaching demonstration evaluation.

After the one-week-long training, after school teachers were randomly assigned to students from 122 grades in 31 schools. In addition, AFF randomly selected two thirds of the grade-schools (for example, 80 grade-schools in the first semester) for the after school tutoring treatment. In the selected grades, a randomly selected one half of the students is eligible for after school tutoring. In this manner, everyone will be eligible at least once during the school calendar year, which consists of three semesters. In addition, students eligible for tutoring are randomly assigned to groups of tutoring class sizes of 4, 6, 8 or 10.

Intervention Start Date

2016-05-18

Intervention End Date

2017-06-30

Primary Outcomes (end points)

The main outcomes are students’ mathematics test scores in the final exams of each semester during the intervention period. As a secondary outcome measure, we will use the test scores of a simple computation test and Raven’s matrix test, as they are directly related to the after school tutoring for mathematics. As a tertiary outcome measure, we will use the subject grades in Chichewa, English, primary science, social studies, art and life skills to explore the potential spill over of the after school tutoring.

To explain the mechanism between after school tutoring and academic achievement, we will examine 1) after school teachers’ inputs (motivation, absenteeism, and effort), 2) existing teachers’ input, 3) parents' input, and 4) students’ motivation, absenteeism, study hours, and non-cognitive traits (self-esteem, Big 5 personality traits) and so on.

To explain the mechanism between after school tutoring and academic achievement, we will examine 1) after school teachers’ inputs (motivation, absenteeism, and effort), 2) existing teachers’ input, 3) parents' input, and 4) students’ motivation, absenteeism, study hours, and non-cognitive traits (self-esteem, Big 5 personality traits) and so on.

Primary Outcomes (explanation)

Secondary Outcomes (end points)

Secondary Outcomes (explanation)

Experimental Design

1. Job advertisements for after school tutors were posted throughout the schools and villages in the catchment area beginning from January 25th, 2015. The NGO advertised vacancies through posters which describe the job and work conditions. High school graduation was the minimum requirement for the position, and applicants were asked to submit an application form along with their Malawi School Leaving Exam transcript (secondary school diploma). Applications were received at 31 primary schools in the catchment area and AFF office. AFF invited applicants to participate in the recruitment meeting in which they participated in the baseline survey, took a math test, and performed a teaching demonstration.

2. The math test consists of 90 questions, which are chosen from 4th to 12th grade textbooks. For the teaching demonstration, applicants were given 2 math problems from the secondary school level (9-10th grade level). Each applicant had 10 minutes to prepare the demonstration and 10 minutes to present it. Several pairs of trained AFF staff members were randomly assigned to the applicants in order to evaluate each applicant’s teaching skill. Specifically, they evaluated overall performance based on the applicant’s engagement with the audience, delivery of mathematical elements in the problems, time management, and enthusiasm.

3. AFF randomly divides applicants into two groups. 173 were assigned to the “math test” group, while the other 174 were assigned to the “teaching demonstration” group. 132 out of 173 in the math test group were hired based on their math test score as after school teachers. 130 out of 174 in the teaching demonstration group were hired based on their teaching demonstration evaluation.

4. Selected after school teachers were required to complete a one-week-long training, during which they were subjectively evaluated by evaluators (regular employees of AFF). Trainees participated in a mock teaching session, where they presented how to solve a pre-assigned math problem from an assigned math course. Evaluators then scored trainees on a scale from 1 to 3 in terms of engagement with the audience, delivery of mathematical elements in the problems, time management, and enthusiasm.

5. The first semester of the experiment is the third semester of the academic year 2015/16 (April 11, 2015 - July 15, 2016). 122 grades (4th-7th grade) in 31 schools were randomized to either tutoring or no tutoring based on students’ grade and average class level computed from students’ math exam score.

Group 1 (42 grades): no tutoring (5,303 students)

Group 2 (78 grades): tutoring (6,396 students). Out of 6,396 students, 2,849 students were randomly assigned and 3,547 students were not assigned to tutoring.

6. As a result of the above process, 2,849 students out of 6,396 students were randomly assigned to after school tutoring in the third semester of the academic year 2015/2016. The remaining students will be eligible for the tutoring in the following semesters.

7. Each teacher was also randomly assigned to 2 out of 414 after school classes. Each class was held twice a week, either Monday-Wednesday or Tuesday-Thursday. The duration of the tutoring program was 7 weeks per semester, and eligible students took 13 classes in total.

8. There are four tutoring class sizes: 4, 6, 8, and 10. These four class sizes were randomly distributed within each school-grade. For example, when one school-grade consists of 35 students in the tutoring treatment, each of the four class sizes (4, 6, 8, and 10) is initially assigned in a random order.

9. Students were then assigned to the classes in descending order of their baseline math scores. As a result, they were in the same class as those with a similar baseline math score.

10. The collaborating NGO commits to hire the after school teachers for at least three semesters. In the second semester of the experiment (i.e. the 1st semester of the academic year 2016/17), 120 grades (5th-8th grade; students moved on to the next grade as the school year changed) in 31 schools were randomized to either tutoring or no tutoring. School-grades which were not in the tutoring group (42 grades) in the first semester of tutoring were automatically included in the tutoring group. Of the school-grades which received tutoring in the first semester (78 grades), 50% (38 grades) were randomly selected for the tutoring group, with the remaining 40 grades assigned to the no tutoring group. The random assignment of tutors, class sizes, and students was conducted in the same manner as that of the first semester. We will continue to provide the tutoring service to students through the third semester of the experiment (the 2nd semester of the academic year 2016/17).

2. The math test consists of 90 questions, which are chosen from 4th to 12th grade textbooks. For the teaching demonstration, applicants were given 2 math problems from the secondary school level (9-10th grade level). Each applicant had 10 minutes to prepare the demonstration and 10 minutes to present it. Several pairs of trained AFF staff members were randomly assigned to the applicants in order to evaluate each applicant’s teaching skill. Specifically, they evaluated overall performance based on the applicant’s engagement with the audience, delivery of mathematical elements in the problems, time management, and enthusiasm.

3. AFF randomly divides applicants into two groups. 173 were assigned to the “math test” group, while the other 174 were assigned to the “teaching demonstration” group. 132 out of 173 in the math test group were hired based on their math test score as after school teachers. 130 out of 174 in the teaching demonstration group were hired based on their teaching demonstration evaluation.

4. Selected after school teachers were required to complete a one-week-long training, during which they were subjectively evaluated by evaluators (regular employees of AFF). Trainees participated in a mock teaching session, where they presented how to solve a pre-assigned math problem from an assigned math course. Evaluators then scored trainees on a scale from 1 to 3 in terms of engagement with the audience, delivery of mathematical elements in the problems, time management, and enthusiasm.

5. The first semester of the experiment is the third semester of the academic year 2015/16 (April 11, 2015 - July 15, 2016). 122 grades (4th-7th grade) in 31 schools were randomized to either tutoring or no tutoring based on students’ grade and average class level computed from students’ math exam score.

Group 1 (42 grades): no tutoring (5,303 students)

Group 2 (78 grades): tutoring (6,396 students). Out of 6,396 students, 2,849 students were randomly assigned and 3,547 students were not assigned to tutoring.

6. As a result of the above process, 2,849 students out of 6,396 students were randomly assigned to after school tutoring in the third semester of the academic year 2015/2016. The remaining students will be eligible for the tutoring in the following semesters.

7. Each teacher was also randomly assigned to 2 out of 414 after school classes. Each class was held twice a week, either Monday-Wednesday or Tuesday-Thursday. The duration of the tutoring program was 7 weeks per semester, and eligible students took 13 classes in total.

8. There are four tutoring class sizes: 4, 6, 8, and 10. These four class sizes were randomly distributed within each school-grade. For example, when one school-grade consists of 35 students in the tutoring treatment, each of the four class sizes (4, 6, 8, and 10) is initially assigned in a random order.

9. Students were then assigned to the classes in descending order of their baseline math scores. As a result, they were in the same class as those with a similar baseline math score.

10. The collaborating NGO commits to hire the after school teachers for at least three semesters. In the second semester of the experiment (i.e. the 1st semester of the academic year 2016/17), 120 grades (5th-8th grade; students moved on to the next grade as the school year changed) in 31 schools were randomized to either tutoring or no tutoring. School-grades which were not in the tutoring group (42 grades) in the first semester of tutoring were automatically included in the tutoring group. Of the school-grades which received tutoring in the first semester (78 grades), 50% (38 grades) were randomly selected for the tutoring group, with the remaining 40 grades assigned to the no tutoring group. The random assignment of tutors, class sizes, and students was conducted in the same manner as that of the first semester. We will continue to provide the tutoring service to students through the third semester of the experiment (the 2nd semester of the academic year 2016/17).

Experimental Design Details

Randomization Method

1. Tutor screening randomization

Randomizations were done by a computer in the office of AFF. Job applicants were not aware of this randomization. It remains unknown to them even after they are hired.

2. Tutor – school randomization:

Randomization was done by a computer in the AFF office in the presence of all tutors hired. 31 schools were grouped into 9 clusters (each containing 3-5 schools) by considering the distance between schools. Tutors submitted 3 preferences about the school they wished to work in, and they were assigned to the clusters based on their preferences. Tutors were randomly assigned to the school and grade within their assigned cluster.

3. Student randomization

Randomization for the tutoring grade was done in the presence of the school principals by a computer. Individual randomization within the tutoring grade was done in the AFF office privately by a computer.

Randomizations were done by a computer in the office of AFF. Job applicants were not aware of this randomization. It remains unknown to them even after they are hired.

2. Tutor – school randomization:

Randomization was done by a computer in the AFF office in the presence of all tutors hired. 31 schools were grouped into 9 clusters (each containing 3-5 schools) by considering the distance between schools. Tutors submitted 3 preferences about the school they wished to work in, and they were assigned to the clusters based on their preferences. Tutors were randomly assigned to the school and grade within their assigned cluster.

3. Student randomization

Randomization for the tutoring grade was done in the presence of the school principals by a computer. Individual randomization within the tutoring grade was done in the AFF office privately by a computer.

Randomization Unit

First, tutoring classes are randomly assigned to grades-schools. Second, students are randomly selected for tutoring within selected grades-schools.

Was the treatment clustered?

Yes

Sample size: planned number of clusters

122 grades in 31 schools (4th-7th graders at the onset of the after school tutoring program). As of September, 2016, they became 5th-8th graders.

Sample size: planned number of observations

13,442 students
The number of students is the number of the students appearing on the roll call of grades 4, 5, 6, and 7. The roll call was recorded before the start of the experiment, during the 1st semester of 2015/16 (September 2015 to December 2015).

Sample size (or number of clusters) by treatment arms

1. First semester

Group 1 (42 grades): no tutoring (5,303 students)

Group 2 (78 grades): tutoring (6,396 students). Out of 6,396 students, 2,849 students were randomly assigned and 3,547 students were not assigned to tutoring.

2. Second semester

Group 1 (40 grades): no tutoring (4,253 students)

Group 2 (80 grades): tutoring (6,464 students). Out of 6,464 students, 1,691 students were randomly assigned and 4,773 students were not assigned to tutoring. Note that there were 78 grades in the first semester of the project. However, in the second semester of the project, there were 80 grades which is the combination of Group 1 (42 grades) about 50% of the Group 2 (38 grades) in the first semester.

3. Third semester

Design would be very similar to that of the first and second semester.

Out of 414 classes offered, 208 classes (1462 students) are taught by tutors who are hired based on the math test and 206 classes (1387 students) are taught by tutors who are hired based on the mock teaching performance.

Group 1 (42 grades): no tutoring (5,303 students)

Group 2 (78 grades): tutoring (6,396 students). Out of 6,396 students, 2,849 students were randomly assigned and 3,547 students were not assigned to tutoring.

2. Second semester

Group 1 (40 grades): no tutoring (4,253 students)

Group 2 (80 grades): tutoring (6,464 students). Out of 6,464 students, 1,691 students were randomly assigned and 4,773 students were not assigned to tutoring. Note that there were 78 grades in the first semester of the project. However, in the second semester of the project, there were 80 grades which is the combination of Group 1 (42 grades) about 50% of the Group 2 (38 grades) in the first semester.

3. Third semester

Design would be very similar to that of the first and second semester.

Out of 414 classes offered, 208 classes (1462 students) are taught by tutors who are hired based on the math test and 206 classes (1387 students) are taught by tutors who are hired based on the mock teaching performance.

Minimum detectable effect size for main outcomes (accounting for sample design and clustering)

IRB

INSTITUTIONAL REVIEW BOARDS (IRBs)

IRB Name

SMU Institutional Review Board

IRB Approval Date

2016-05-27

IRB Approval Number

IRB-16-046-A053(516)

IRB Name

Cornell Institutional Review Board for Human Participants

IRB Approval Date

2016-04-27

IRB Approval Number

1604006306

Post Trial Information

Is the intervention completed?

No

Is data collection complete?

Data Publication

Is public data available?

No

Program Files