Evaluating the Design and Impact of the Secondary School Teacher Training Initiative under the Government of Nepal’s School Sector Development Program

The main teacher training (TT) intervention requires all 9th and 10th grade teachers in government schools to attend government-run in-service teacher training modules that are intended to raise their subject knowledge and to motivate and equip them to use practical, demonstration-based teaching methods rather than more traditional teaching methods. The training includes a 10-day session at an Education Training Center (ETC), and then completion by participating teachers of the equivalent of five days of “self-study project work,” which includes independent lesson plan development and other classroom research and practice activities. The TT intervention in study schools differs slightly from the broader intervention to be rolled out throughout Nepal over the next several years in several respects. First, rather than waiting for teachers and schools to request trainings, the ETCs specifically invited teachers in treatment schools, and were asked not to invite teachers in control schools or other schools within the same small geographic areas (associated with the local Village Development Committees) as the control schools. Second, while in the broader roll-out priority will be given to inviting teachers with permanent positions who had not received training under the previous education plan (the School Sector Reform Program or SSRP), ETCs were requested to invite all teachers of grade 9 and 10 math and science, regardless of contract type or previous training experience. Third, while in principle the full SSDP training will include two modules, each including 10 days of training at an ETC and five days of self-study project work, in practice only the first module has so far been rolled out, and only this first module is included in this evaluation.
The supplementary Video Assignment (VA) treatment requires each trained teacher to submit (before he or she can receive full credit for the training) a video of himself or herself (during an entire class session) implementing one of the new lesson plans that he or she is expected to create as part of the self-study project. The aim of adding the VA is to increase teachers’ motivation for investing serious effort in the self-study project activities, which may be important for translating what teachers learn at the ETC into new and improved classroom practices. In what follows, “TT treatment” will refer to the provision of the basic teacher training intervention without the video assignment, while “TTVA treatment” will refer to the provision of the teacher training intervention with the video assignment.

2017-12-19

2018-07-16

Math and science test scores for students in grades 9 and 10. Also, a subset of items on these tests will be generated for those items for which the content was emphasized in the training.

• School-level averages of teachers’ scores on subject knowledge assessments (implicit in the teacher evaluations of the student assessment items), estimated using IRT methods (with raw scores as robustness check), separately for math and science.
• Teacher attendance measures:
o Teacher attendance on the first day of the school visit, and over the previous several days, as recorded in school logbooks
o Teacher attendance on the first day of visit, as recorded in the first question on the teacher questionnaire
o Median of students’ report regarding how frequently the teacher is absent (5-point scale)
• Student attendance:
o Student attendance rate on the most recent full day of classes before today, as reported by head teacher, separately for grade 9 and grade 10 students
• From Head Teacher reports on individual teachers:
o The teacher’s command over math or science subject matter (5-point scale)
o The teacher’s interest is in learning ways to teach more effectively (4-point scale)
o Whether the teacher has ever created from local resources any teaching materials – such as models of 3-dimensional shapes, atoms or the solar system – for use in a grade 9 or 10 math or science course
o Frequency of teacher’s use of teaching materials or other visual aids (other than the chalk board) to help explain concepts (4-point scale)
o Whether teacher has ever collected information from local residents outside the school (such as prices in local markets or interest rates offered by local bankers), or required his students to collect such information
o Frequency of teacher requiring students to work together in small groups (4-point scale)
o Frequency of teacher requiring students to work on longer-term project, for which they must gather information and make practical application (4-point scale
• From teacher questionnaires:
o Self report of minutes spent preparing per class during the most recent full week of classes (calculated from reports of total number of classes and total minutes spent preparing, whether during other class periods or outside of class time).
o Self report of whether teacher uses a written lesson plan as a primary guide while conducting a class
o Self report of how often teacher has students work together in small groups (5-point scale)
o Self report of how often teacher uses examples or homework problems involving local information (7-point scale)
o Opinion regarding importance of using examples involving local information (3-point scale)
o Self report of how often teacher has required students to collect local information, whether by interviewing family or community members, observing family or community activities, or taking measurements (for example, of weather conditions)? (7-point scale)
• From student questionnaire, median responses, separately for math and science teachers
o How often teacher gives homework (5-point scale)
o How often teacher checks student’s homework (5-point scale)
o How often teacher returns student’s homework with corrections (5-point scale)
o How often teacher uses class time for asking questions of any students or holding discussions or interactions about math/science concepts with any students (5-point scale)
o How often teacher uses class time to ask YOU (the student) a question or engaging YOU in a discussion of interaction about math/science concepts (5-point scale)
o How often teacher requires the student to work together in small groups (5-point scale)
o How often teacher uses demonstrations involving physical objects made from local materials or other visual aids to help students understand math/science concepts (5-point scale)
o How often teacher uses demonstrations involving diagrams, pictures or information from the internet (5-point scale)
o How often teacher gives quizzes to students (5-point scale)

The study collects data from 204 schools. The primary randomization divides the 204 study schools into two groups of equal size: 1. 102 “Phase I” schools, which were to receive the SSDP teacher training in late 2017; and 2. 102 “Phase II” schools, which were to receive the SSDP teacher training only after May of 2019, and which serve as the control group during the period of study. To minimize the potential for spillover effects of training on Phase II study schools, other schools in the same small geographic areas that contain the Phase II schools will also receive the relevant training only after May of 2019. The secondary randomization divides the 102 Phase I schools into two groups: 1. 51 TTVA schools, in which each teacher must submit a video of himself or herself implementing in his or her classroom a new lesson plan developed as part of the SSDP training to receive full credit for the training; and 2. 51 TT schools, in which no video is required to receive full credit for the training.

To achieve a sample representative of most of Nepal, while containing costs, the research team chose to stratify the sample first by district. To reduce data collection costs further, the team (in consultation with its partners in the Nepalese government) eliminated from consideration 10 of the most remote or otherwise difficult districts. From the remaining 65 districts (which contain 94.3% of Nepal’s schools), the team randomly selected a representative set of 16 districts, and then sampled schools only within those districts.

At the request of the team’s government partners, and to increase the study’s power to identify impact on the teachers for whom the TT training was most likely to have impact, the team sorted schools within districts into two strata: “priority” and “non-priority” schools, and over-sampled the former. Schools were identified as priority schools if there was no evidence (in hard copy records made available by the National Center for Educational Development) of any permanent teacher, or any teacher for which contract type was unknown, having completed all three modules of the SSRP training. The specifics of this rule were dictated largely by the idiosyncrasies of the only existing records identifying which schools and teachers had received SSRP training. The team chose to select two-thirds of the sample within a district from the priority stratum, while the other one-third would come from the non-priority stratum.

Schools were randomized, after stratification, using random numbers drawn by the research team.

School

Yes

204 schools

8,000 students, and about 400 teachers and 204 head teachers.

102 control schools, 51 schools with TT treatment, and 50 schools with VA treatment.

For a power level of 0.8, at the 95% significance level we should be able to detect an increase in test scores of 7 percentage points, which is about 0.3 standard deviations of the distribution of test scores.

Evaluating the Design and Impact of the Secondary School Teacher Training Initiative under the Government of Nepal’s School Sector Development Program