Impact and Mechanisms: Why Consulting Matters in Human Capital Intensive Organizations. Evidence from a Field Experiment on Teacher Coaching.

As part of this study, we aim to measure the impact of a teacher-coaching intervention. We will do so in a context where all teachers (whether or not they receive the coaching intervention) will benefit from a professional development course designed to improve their teaching skills. Half of those teachers will be selected to benefit from a coaching intervention as well.

3.1 Professional development course
We will implement an intervention where all participating teachers and coaches attend five one-day course modules on the following education production factors: i) Physical surroundings, ii) Structuring schoolwork, iii) Self-regulation iv) The teacher's focus, and v) The teacher's communication. As part of these course modules, teachers will learn to use both technological input factors (e.g. timer, checklist, five-point scale, etc.) and behavioral input factors (e.g. support in transition phases, use calm voice, mentalizing, etc.). Furthermore, during each module, teachers will be asked to reflect on their own students and to set an agenda for themselves detailing how they intend to change their teaching practices so as to take into account what they will have learnt during the course modules. In addition, each course module will start out with rehearsal and reflection on what they have learnt in order to consolidate knowledge.

The course modules will be held on the following dates:
- Course module 1: August 9th, 2017
- Course module 2: September 14th, 2017
- Course module 3: October 31st, 2017
- Course module 4: November 28th, 2017
- Course module 5: February 7th, 2017

3.2 Treatment: Teacher-coaching program
Teachers selected to receive the teacher-coaching program will be paired with a co-teacher, their "coach", with whom they will spend 2.5 hours per week during an entire school year. Co-teachers will help teachers get closer to the production-possibility frontier through "peer coaching", as part of which the teacher receives feedback on how to improve teaching practice. These sessions will be dedicated to the implementation of what will have been covered during the professional development courses.

2017-08-09

2018-06-30

In order to answer our research questions, several rounds of data collection will be carried out (for each batch) at different points in time:
1/ At the beginning of the school year, during the first 20 minutes of the first course module (baseline data collection, August 2017 - before the results of the randomization are shared with the schools)
2/ During the first 20 minutes of each remaining course module (follow-up surveys)
3/ After the first coaching session carried out after each course module (monitoring data collection)
4/ At the end of the school year, upon completion of the course modules (endline data collection, June 2018)

The data will be gathered through different data sources:
- Administrative registers: student, parent and teacher background, tracking of students and teachers, output measures (e.g. national test scores and grades for the teacher's past, current, and future students)
- Observation data: classroom instrument developed from the course material and/or CLASS-UE measurement
- Teachers and students surveys: they will be collected online using the SurveyXact platform

Monitoring data
We will collect information depicting how the intervention is carried out. In particular, we will collect information on the identity of the teachers attending the professional development course and on those actually receiving the coaching intervention.
Furthermore, descriptive information will also be collected so as to understand how the coaching intervention is implemented. Teachers will be asked to answer (together with their coach) a short questionnaire after the first coaching session carried out after each course module. This questionnaire will collect information on their experience with the co-teaching intervention.

Outcome variables
Our primary outcome is teachers' teaching skills and, more specifically, whether or not they use the elements taught during the professional development course in class. This will be assessed by surveyors who will attend teachers' class and will rate the extent to which they use each of the techniques taught during the course. We will supplement these class observations by a measure of teaching skills as reported by the students using a validated index of teaching quality (TRIPOD).

Our secondary outcome is students' achievements and, in particular, their reading skills (as measured by national standardized reading tests) and socio-emotional skills (as measured by national standardized wellbeing surveys).

As there are strong reasons to expect significant selection with respect to the type of teachers who enroll in a coaching program, we will measure its impact using a randomized controlled trial design.

The coaching intervention will be randomized at the teacher-level. For implementation-related reasons, it was decided that the draw would be stratified by school. As a consequence, we will randomly select in each school half of the language arts teachers to benefit from the coaching intervention. In schools where an uneven number of teachers will be enrolled in the experiment, the number of teachers who should benefit from the coaching intervention was decided in advance for logistical reasons. We will adjust for the ensuing variation in teachers' probability to receive the coaching intervention across schools in our estimation strategy.

When schools will have appointed more than one coach, the teacher-coach pairs will be randomly generated.

Randomization done in office by a computer.

The coaching intervention will be randomized at the language arts teacher-level. Again, for implementation-related reasons, it was decided that the draw would be stratified by school.

Yes

17 schools

68 teachers 1490 students

In each school, 50% of the teachers enrolled in the experiment will benefit from the intervention.

Our power analysis suggests that with a power of 80% we would be able to detect a minimum effect size of 0.50 to 0.60 standard deviations in teachers' behavior and a minimum effect size of 0.23 to 0.30 standard deviations in students' test scores.

Calmar et al. (2017). Analysis plan