AEA RCT Registry

Trial Information

General Information

Title

Can Teacher Peer Feedback Improve Test Scores? Evidence from an RCT in 182 Schools

RCT ID

AEARCTR-0001779

Initial registration date

November 24, 2016

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published

November 24, 2016, 9:16 AM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Country

United Kingdom of Great Britain and Northern Ireland

Region

countrywide

Primary Investigator

Name

Felix Weinhardt

Affiliation

Contact Primary Investigator

Other Primary Investigator(s)

PI Name

Richard Murphy

PI Affiliation

Contact Investigator

Additional Trial Information

Status

On going

Start date

2013-05-01

End date

2017-05-01

Keywords

Education

Additional Keywords

teacher training, teacher feedback, lesson study, student performance

JEL code(s)

Secondary IDs

Abstract

It is widely accepted that teachers are the most important factor in school effectiveness. However, there is little robust quantitative evidence of the effectiveness of programmes that improve teacher quality. Of reliable research carried out on teacher development, knowledge based training has typically been found to be ineffective, while programmes that involve teacher observations that are embedded into the school alongside effective feedback have been found to have effects (Taylor and Taylor, 2013).
Lesson Study is a programme incorporating teacher observations and feedback. It is a professional development programme with a long history of use in Japan and is being increasingly used in the UK. Teachers work in small groups to plan lessons that address a shared teaching and learning goal. They then observe each other’s lessons, focusing on pupil learning rather than the teacher. They then discuss the lesson, refine the lesson plans and repeat the process. In the planning, two 'case pupils' are selected that are typical of a group of pupils in the class, so that the impact of the lessons on pupils with identified barriers to learning can be monitored. After the lesson has taken place, it is discussed, analysed and evaluated by the group of teachers, leading to revisions being made in content, structure and delivery. This process includes feedback from the case pupils in the form of a teacher/pupil interview. The knowledge generated by the whole process is then shared more widely with other colleagues.
An evaluation of the UK’s National Strategies’ Leading Teachers Programme, which involved Lesson Study, showed that those schools using this approach (among others) out-performed a comparison group in both English and Mathematics. Lesson Study also shares many of the key characteristics of effective CPD that were identified in a systematic review produced by the Evidence for Policy and Practice Information and Co-ordinating Centre. Research investigating the National Strategies’ Leading Teachers Programme (Hadfield, Jopling and Emira, 2011), of which Lesson Study was a key element, revealed positive impacts on pupil outcomes. However, this study did not take account of potential differences in the treatment and control groups used in the study, nor was it able to follow up the long-term impacts of the programme.
The trial of Lesson Study incorporated Talk for Learning, an approach which aims to improve the quality of classroom talk in order to increase pupils' engagement, learning and attainment. Participating schools were asked to deliver Lesson Study within literacy and maths lessons using Talk for Maths and Talk for Literacy with teachers trained in specific approaches for both. The trial was therefore a test of the effectiveness of this method as well as of the Lesson Study approach.

External Link(s)

Registration Citation

Citation

Murphy, Richard and Felix Weinhardt. 2016. "Can Teacher Peer Feedback Improve Test Scores? Evidence from an RCT in 182 Schools." AEA RCT Registry. November 24. https://doi.org/10.1257/rct.1779-1.0

Former Citation

Murphy, Richard and Felix Weinhardt. 2016. "Can Teacher Peer Feedback Improve Test Scores? Evidence from an RCT in 182 Schools." AEA RCT Registry. November 24. https://www.socialscienceregistry.org/trials/1779/history/11998

Sponsors & Partners

Experimental Details

Interventions

Intervention(s)

It is widely accepted that teachers are the most important factor in school effectiveness. However, there is little robust quantitative evidence of the effectiveness of programmes that improve teacher quality. Of reliable research carried out on teacher development, knowledge based training has typically been found to be ineffective, while programmes that involve teacher observations that are embedded into the school alongside effective feedback have been found to have effects (Taylor and Taylor, 2013).
Lesson Study is a programme incorporating teacher observations and feedback. It is a professional development programme with a long history of use in Japan and is being increasingly used in the UK. Teachers work in small groups to plan lessons that address a shared teaching and learning goal. They then observe each other’s lessons, focusing on pupil learning rather than the teacher. They then discuss the lesson, refine the lesson plans and repeat the process. In the planning, two 'case pupils' are selected that are typical of a group of pupils in the class, so that the impact of the lessons on pupils with identified barriers to learning can be monitored. After the lesson has taken place, it is discussed, analysed and evaluated by the group of teachers, leading to revisions being made in content, structure and delivery. This process includes feedback from the case pupils in the form of a teacher/pupil interview. The knowledge generated by the whole process is then shared more widely with other colleagues.
An evaluation of the UK’s National Strategies’ Leading Teachers Programme, which involved Lesson Study, showed that those schools using this approach (among others) out-performed a comparison group in both English and Mathematics. Lesson Study also shares many of the key characteristics of effective CPD that were identified in a systematic review produced by the Evidence for Policy and Practice Information and Co-ordinating Centre. Research investigating the National Strategies’ Leading Teachers Programme (Hadfield, Jopling and Emira, 2011), of which Lesson Study was a key element, revealed positive impacts on pupil outcomes. However, this study did not take account of potential differences in the treatment and control groups used in the study, nor was it able to follow up the long-term impacts of the programme.
The trial of Lesson Study incorporated Talk for Learning, an approach which aims to improve the quality of classroom talk in order to increase pupils' engagement, learning and attainment. Participating schools were asked to deliver Lesson Study within literacy and maths lessons using Talk for Maths and Talk for Literacy with teachers trained in specific approaches for both. The trial was therefore a test of the effectiveness of this method as well as of the Lesson Study approach.

Intervention Start Date

2013-09-02

Intervention End Date

2015-07-20

Primary Outcomes

Primary Outcomes (end points)

Student performance (for details see uploaded SAP document)

Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)

Secondary Outcomes (explanation)

Experimental Design

In order to have sufficient schools to meet the power calculations the trial was designed to maximise sample size by keeping costs to a minimum. This was achieved by relying on Key Stage 2 (KS2) test score data from the treatment and control schools. Once a school was assigned as a control school they received a letter stating that as a result of the randomisation process they had not been selected, and we had no further contact with them for the duration of the experiment. We are still able to collect outcome data on the students from the control schools by using the national pupil database.
Schools that were selected into the treatment group were asked to select three teachers to be part of the Lesson Study trial. These teachers were all to come from years 4 and 5. Where that was not possible, because there was only one class per cohort for example, schools were free to involve teachers from Years 3 or 6. This means that is possible for students to be taught by Lesson Study teachers from Year 3 through to Year 6.
Table 1 below shows the targeted cohorts in academic Years 4 and 5 during the calendar school years 2013-14 and 2014-15. The outcome is measured at the end of Year 6. C1 and C2 sat their respective KS2 exams in the summer of 2015-16 and 2015-16 respectfully. C1 will only have one year of treatment, C2 will likely have two years of treatment, but both will have a year without treatment before their KS2 examinations. However, for reasons explained above it will also be the case that some students will be treated in schools during Year 6. These students could have taken their KS2 examinations in 2013-14, 2014-15, or 2015-16. Moreover there are likely to be students taught by Lesson Study Teachers, for whom we would not have observed their KS2 outcomes by the summer of 2015-16.

Table 1
2013-14 2014-15 2015-16 2016-17 2017-18
Year 2 C4 (KS1 exam)
Year 3 C3 C4
Year 4 C2 C3 C4 .. ..
Year 5 C1 C2 C3 C4 ..
Year 6 C0 C1 C2 C3 C4 (KS2 exam)

Therefore we will be collecting outcome measures for three cohorts of students (C0, C1 and C2) in both the treatment and control schools. We will also collect these students KS1 test scores from the NPD to use as baseline measures of achievement. Cohorts C1, C2 and C3 all sat their KS1 exams before the intervention took place, so these test outcomes can serve as baseline even in the presence of across-year spillovers of the treatment within schools.

Experimental Design Details

Randomization Method

Randomization done in office by a computer setting different seeds

Randomization Unit

Schools

Was the treatment clustered?

Yes

Experiment Characteristics

Sample size: planned number of clusters

160 schools

Sample size: planned number of observations

160 schools times number of students: exact number unknown since the trial targets teachers and we only get access to baseline as well as outcome data through administrative national exams. There is no baseline or outcomes data collection done by ourselves. Instead, this information is gathered through ex-post linkeage with administrative test score records.

Sample size (or number of clusters) by treatment arms

This resulted in a total of 89 schools assigned to treatment, 92 to control and one was left unassigned.
Details see uploaded consort diagram in analysis plan.

Minimum detectable effect size for main outcomes (accounting for sample design and clustering)

The study only detects true effect sizes of 0.1 s.d. in the outcome with a probability of about 80 percent. Details in uploaded SAP

Supporting Documents and Materials

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

IRB

Institutional Review Boards (IRBs)

IRB Name

LSE Research Governance Board

IRB Approval Date

2016-05-25

IRB Approval Number

273-472

Analysis Plan

Can Teacher Peer Feedback Improve Test Scores? Evidence from an RCT in 182 Schools

Pre-Trial

General Information

Locations

Primary Investigator

Other Primary Investigator(s)

Additional Trial Information

Registration Citation

Interventions

Primary Outcomes

Secondary Outcomes

Experimental Design

Experiment Characteristics

Institutional Review Boards (IRBs)

Post-Trial

Study Withdrawal

Intervention

Data Publication

Program Files

Relevant Paper(s)

Reports & Other Materials