Improving Accountability in Khyber Pakhtunkhwa's Schools

The study sample of 240 schools consists of 20 circles (5 girls' circles and 15 boys' circles) with 12 schools in each circle, across three districts. Treatment was randomly assigned within circles. In each circle, there are 4 schools in the ATE treatment, 4 in the SIR treatment, and 4 in the 'business-as-usual' control.

ATE Intervention (80 schools)

All schools will receive three 'surprise' visits from an independent district inspector during the 2017/8 school year. The matching of inspectors to schools, and the dates of the site visits will be determined randomly by researchers. Inspectors will be provided with information on the school the evening before the visit, and will be instructed not to notify the school. If the inspector cannot visit that school that day, he/she will need to provide a reason and, if this is acceptable, he/she will be reassigned an alternative, randomly selected school the following day. All inspectors will be provided with: the ATE protocol; a tablet (pre-loaded with a relevant computer assisted personal interview (CAPI) program with which the ATE will be conducted, including teacher/student rosters) and tripod stand; a paper teacher pedagogy assessment tool; and a Wi-Fi box with credit to upload data. Upon arrival at the start of the school day, the inspector will explain the new ATE system to the head teacher and find out when the Grade 4 math class will be taught. Each inspector will then collect information on four performance measures: the presence and pedagogy of the Grade 4 math teacher, together with the presence of and learning outcomes for his/her students in the Grade 4 math class.

Scoring rubric. These four performance measures will be scored to produce an overall ATE. For the first three, scoring is based on absolute performance against a pre-specified objective rubric.

" Teacher attendance: present or excused absent = 8 points, unexcused absent = 0 points.
" Student attendance: sliding scale from 0 to 8 points for share of students present (max for >90% present).
" Teacher pedagogy: sliding scale from 0 to 8 points for average share of students engaged in active teaching activities (max for >88% engaged).

For student learning, scoring is based on relative performance and is intended to capture the main features of the 'pay-for-percentile' approach (Barlevy and Neal 2012). Specifically, once all data have been collected from all 80 ATE study schools, Grade 4 math teachers will be put into 'bins' of 8 based on their start of year (first visit) 'percentage of 50 Grade 4 math questions answered correctly' score. At the end of the year (third visit), these teachers will be ranked within bins. The Grade 4 math teacher in the school with the top rank will receive 8 marks, the next rank 7 marks, down to zero for the Grade 4 math teacher in the school with the bottom rank. This scoring of student learning will be done by the research team.

The detailed protocol for all four dimensions of this scoring rubric will be communicated to inspectors during an October 2017 training (following a September 2017 baseline survey, but before treatment begins), and to the Grade 4 math teacher at the start of the first surprise visit, via a letter outlining the intervention and the scoring.

Career incentives. The scores across these four performance measures will be aggregated into a single, cardinal score for each Grade 4 math teacher. Specifically, the overall score will be the sum of 10 measures (teacher attendance x3, student attendance x3, teacher pedagogy x3, student performance x3) scored on the 0-8-point scale. (While there will be a single measure of student performance, it will be included three times in the score calculation to ensure that the four categories are weighted equally as teachers were led to expect in the information sheet). These cardinal scores will feed into an ordinal ranking exercise. Within each of the 20 circles in the study, there are 4 schools randomly assigned to the ATE treatment. At the end of the school year, the four Grade 4 math teachers within each circle x ATE treatment pair will be ranked on the basis of their cardinal ATE score. In the event of a tie, the inspector (who will have visited all schools in the comparison set) will be asked to consider all four dimensions of performance and break the tie in favour of one teacher or the other. The top-ranked teacher will have his/her promotion fast-tracked by one-year, while the bottom-ranked teacher will have his/her promotion delayed by one-year. The two teachers in the middle of the ranking will experience no change.

In an attempt to prevent dysfunctional behaviour (collusion and/or demotivation) among teachers, the precise details of the promotions process will not be communicated to teachers. Details are provided in the ATE information sheet.

Audits. To incentivise inspectors to undertake the ATE thoroughly and objectively, the scores submitted will be audited by individuals with an arms-length relationship to the inspectors/schools. There are two auditor treatments: in the first auditors are drawn from a pool of district officials from non-study districts, and in the second auditors are drawn from a pool of secondary school teachers from study districts. Each inspector will have one of their 'surprise visits' assigned to the district official treatment, and another to the secondary school teacher treatment. Both types of auditor will review all of the information collected during the visits (photographic proof of attendance, video classroom observations) and will assign their own ATE score. Discrepancies between inspector and auditor scores will be noted. During training, inspectors will be told that their inspections will be audited and that their performance will be recognized and rewarded in two ways. First, inspectors who have performed well will be awarded a certificate at a public ceremony at the end of the school year. Second, inspector performance will be included in the dossier that forms part of their own Annual Confidential Report, and could therefore influence decisions about their own promotion and salary increments.

Marks assigned by the auditors will also be verified by the study team, as part of the process of evaluating the efficacy of the two different types of auditor. The auditor will be told this, but will not receive any other form of incentive.

SIR Intervention (80 schools).

The SIR intervention will be identical to the ATE intervention except that each inspector will collect information on the presence of the head teacher, all teachers, and all students, together with the pedagogy of two teachers.

Scoring rubric. These four performance measures will be scored to produce an overall SIR score. In each case, scoring is based on absolute performance against a pre-specified objective rubric.

" Head teacher attendance: present or excused absent = 8 points, unexcused absent = 0 points.
" Teacher attendance: sliding scale from 0 to 8 points for share of teachers present (max for >90% present).
" Student attendance: sliding scale from 0 to 8 points for share of students present (max for >90% present).
" Teacher pedagogy: sliding scale from 0 to 8 points for average share of students engaged in active teaching activities (max for >88% engaged) across the two classroom observations.

The detailed protocol for all four dimensions of this scoring rubric will be communicated to the inspector during training, and to the head teacher at the start of the first surprise visit via an information sheet.

Career incentives. The scores across these four performance measures will be aggregated into a single, cardinal score for each head teacher; specifically, the overall score will be the unweighted sum of 12 measures (head teacher attendance x3, teacher attendance x3, student attendance x3, teacher pedagogy x3) scored on the 0-8-point scale.

As in the ATE intervention, these cardinal scores will feed into the ordinal ranking exercise within circles. At the end of the school year, the schools within each circle x SIR treatment pair will be ranked on the basis of their cardinal SIR score. In the event of a tie, the inspector (who, by construction, visited all schools in the comparison set) will be asked to consider all four dimensions of performance and break the tie in favour of one school or the other. The head teacher of the top-ranked school will have his/her promotion fast-tracked by one-year, while the head teacher of the bottom-ranked school will have his/her promotion delayed by one-year. The two head teachers of the two middle-ranked schools will experience no change.

Again, in an attempt to prevent dysfunctional behaviour (collusion and/or demotivation) among head teachers, the precise details of the promotions process will not be communicated to head teachers. Details are provided in the SIR information sheet.

Audits. Scores awarded as part of the SIR will also be audited, following an identical protocol to the ATE intervention.

'Business-as-usual' control (80 schools)

The same district inspector will be visiting schools under different treatments. This creates the possibility of spillovers from the ATE and SIR interventions into controls schools. To mitigate this, at the start of inspector training (before any mention of the ATE and SIR interventions) inspectors will take part in facilitated group discussions about the protocols currently used for teacher 'annual confidential reports', and for school inspections, in their district. On the basis of these discussions, a written 'business-as-usual' protocol will be established for each district-gender. During training it will be stressed to each inspector that it is critically important that he/she adheres to this 'business-as-usual' protocol for all schools in the district that have not been included in either the ATE or SIR interventions.

2017-11-07

2018-02-10

Our primary outcomes relate to students. The first is student learning, as measured by the difference in performance of students in Grade 4 math classes at endline compared to baseline on written, independently administered math tests. The second is student drop out, measured as a student having been enrolled in the class at baseline but not at endline. The third is student attendance, as measured on the day (and day before) the baseline and endline surveys.

Student learning. Grade 4 math students in all 240 schools will sit a written math test at baseline in September 2017 and again at endline in March 2018. These test scores will be used to obtain estimates of student learning (using item response theory), which will be used to test for treatment impacts in a (now standard) ANCOVA student-level specification.

Student drop out. We define drop outs as students who enroll at the start of the school year but either stop attending during the year and are withdrawn from the register or fail to enroll again the following school year (the latter outcome being conditional on funding availability for a second follow-up).

Student attendance. We measure student attendance on the day of the baseline survey as well as on the previous day. We again collect same day and previous day attendance at endline. These provide us with two measures of attendance.

Our secondary outcomes relate to teacher and headteacher effort, namely head teacher attendance and grade 4 math teacher attendance as well as teaching practices as reported by students.

Head teacher attendance. Head teacher attendance is recorded on the day of the (unannounced) baseline and endline surveys.

Teacher attendance: During surveys, enumerators will ask the head teacher for access to the official teacher register. From this they will record whether the teacher was present, absent but sanctioned, or absent but unsanctioned on each of the previous 5 school days. If there is no record, the enumerator will ask the head teacher to provide a response. Our measure of grade 4 math teacher attendance will average over these 5 days. (This is a different measure of attendance from those collected during ‘surprise visits’ in ATE and SIR schools as part of the intervention.)

Teaching practices. These will be reported by children – indicator variables for whether teacher provides help outside the classroom, whether teacher provides feedback to their parents, and whether the teacher splits students into groups for activities in class (averaged over students in the teacher’s class since the regression will be at the teacher level).

Head teacher helps with grade 4 class. To determine whether the interventions generated more of a focus on grade 4 (that was the class that was tested), we will also examine the effects of the interventions on a dummy variable for whether the head teacher helped with teaching the grade 4 class.

This study will test whether (either of) two new school accountability systems can improve student learning outcomes in rural, public primary schools in KP province, Pakistan. The core of the experimental design is a school-level randomized control trial of either an Annual Teacher Evaluation (ATE), or a School Inspection Report (SIR), against a ‘business-as-usual’ control. These three systems are described in detail under the intervention section above. The trial will take place in 240 schools within 20 circles (school districts) during the 2017/18 school year. In each circle, 4 schools will be randomly assigned to one of three groups: ATE, SIR, or ‘business-as-usual’ control. Both the ATE and the SIR will be based on surprise visits by independent district inspectors, who will collect information on the presence of staff and students, pedagogy of teachers, and performance of students in math classes using electronic tablets. Inspectors will award scores based on this information which will then determine teacher and head teacher promotions through tournaments within circles. To incentivise inspectors to do a thorough and objective job, the electronic records of this information will be audited, with the audit reports placed on file to determine the inspectors’ own promotions. A secondary feature of the experimental design is an inspector-level randomized control trial of alternative forms of auditing, either by officials from non-study districts, or by secondary school teachers within the same district.

Assignment of schools to treatment (ATE, SIR, control), and assignment of auditors to inspectors (official from non-study district or secondary school teacher) will be done in office, by computer by the PI team.

Other assignments required as part of the intervention (e.g. choice of classroom to visit for teacher pedagogy observations under the SIR treatment) will be done in the field, automatically by the tablet on which the inspection is being undertaken using data entered by the inspector upon arrival at the school.

Accountability treatment: the unit of randomization is a school.
Auditor treatment: the unit of randomisation is an inspection visit.

Yes

Sampling of 240 schools. Following discussions with the E&SE Department, three districts (Charsadda, Mardan, and Nowshera) were identified as locations where it would be logistically feasible to operate. Enrolment data from Pakistan’s Independent Monitoring Unit identified 1,547 schools satisfying the following criteria:

• Rural, public, primary school
• Grade 4 enrolment between 10-50 inclusive (based on Fall 2016 enrolment)
• Grade 4 teacher neither up for promotion nor transfer within next two years, not the head teacher, and not having the Pakistan Reading Program (PRP) in place.

Dropping schools used in the same district as a pilot, leaves 870 schools for sampling—629 males and 241 female. These schools are organised into circles, which are administrative units that operate much like a school district; within each circle, there is a set of boys’ schools and a set of girls’ schools, each with separate E&SE department officials overseeing their administration. Accordingly, there are two circle-genders within each circle—one boys’ circle-gender and one girls’ circle-gender.

A total of 20 circle-gender pairs will be drawn from the 48 circle-gender pairs available. To be drawn, a circle-pair must satisfy the requirement that there are at least 18 schools (to allow for replacements if needed). Given the relative scarcity of girls’ schools, this requirement will result in 5 girls’ circles and 15 boys’ circles. Within each circle-gender pair, 12 schools will then be drawn for inclusion in the study. 12 schools x 20 clusters gives the 240 schools in the study.

6,240 students. 240 teachers. 240 headteachers. 40 inspectors.

The ‘accountability’ treatment will be assigned randomly across the 12 schools within each of the 20 circle-genders: 4 schools in the ATE treatment, 4 in the SIR treatment, and 4 in the business-as-usual control. In total, there will therefore be 80 Annual Teacher Evaluation (ATE) schools, 80 School Inspection Report (SIR) schools, and 80 Business-as-usual Control schools.

There will also be an ‘auditing’ treatment. For each inspector, one visit will be randomly assigned to audit by an official from a non-study district and another visit to audit by a secondary school teacher from the same district.

Power calculations using math test scores from the Learning and Educational Achievement in Punjab Schools (LEAPS) study in Punjab, Pakistan show that we can detect and effect size of 0.125 standard deviations over baseline achievement with 80 schools in each of the two treatment groups and 80 schools in the control group.

Pre-analysis Plan