Management and Student Achievement: Evidence from a Randomized Field Experiment

Last registered on August 30, 2017

Pre-Trial

Trial Information

General Information

Title
Management and Student Achievement: Evidence from a Randomized Field Experiment
RCT ID
AEARCTR-0002377
Initial registration date
August 30, 2017

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
August 30, 2017, 2:18 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Primary Investigator

Affiliation
Harvard University

Other Primary Investigator(s)

Additional Trial Information

Status
Completed
Start date
2014-02-01
End date
2017-05-01
Secondary IDs
Abstract
This study examines the impact on student achievement of implementing management training for principals in traditional public schools in Houston, Texas, using a school-level randomized field experiment. Across two years, principals were provided 300 hours of training on lesson planning, data-driven instruction, and teacher observation and coaching. The findings show that offering management training to principals significantly increases student achievement in all subjects in year one and has an insignificant effect in year two. We argue that the results in year two are driven by principal turnover, coupled with the cumulative nature of the training. Schools with principals who are predicted to remain in their positions for both years of the experiment demonstrate large treatment effects in both years - particularly those with principals who are also predicted to implement the training with high fidelity - while those with principals that are predicted to leave have statistically insignificant effects in each year of treatment.
External Link(s)

Registration Citation

Citation
Fryer, Roland. 2017. "Management and Student Achievement: Evidence from a Randomized Field Experiment." AEA RCT Registry. August 30. https://doi.org/10.1257/rct.2377-1.0
Former Citation
Fryer, Roland. 2017. "Management and Student Achievement: Evidence from a Randomized Field Experiment." AEA RCT Registry. August 30. https://www.socialscienceregistry.org/trials/2377/history/21005
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
Year 1: Two (nonconsecutive) weeks of training (summer 2014), ongoing coaching and professional development from Chief Management Officer, set of high-quality interim assessments. 170 total hours of training (81% on content, 19% on systems).

Year 2: One-week training (summer 2015), ongoing coaching and professional development from Chief Management Officer, data monitoring systems provided by HISD to all schools, set of high-quality interim assessments. 130 total hours of training (38% on content, 62% on systems)

Principals and other applicable school leaders were expected to collect teacher lesson plans and provide feedback before plans are implemented, observe teachers in the classroom and give feedback at least biweekly, and work with teachers to analyze student data and create action plans.
Intervention Start Date
2014-05-15
Intervention End Date
2016-06-22

Primary Outcomes

Primary Outcomes (end points)
Percentage of trainings attended by principals, observations/coaching sessions per teacher; survey measure of perceived effectiveness of training, high-stakes test scores, low-stakes test scores
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
The intervention relied on three levers of school management for the treatment to impact student test scores, the primary outcome of interest. The first lever was instructional planning. Teachers in treatment schools were to turn in lesson plans to principals, or another designated school official, before teaching the lesson in order to receive feedback in one-on-one coaching sessions. Principals and school leaders received training to identify well-made and poorly-made lesson plans and to effectively communicate ways to improve lesson plans with teachers. The second lever was data-driven instruction. Students were assessed every 6-8 weeks on interim assessments developed for this intervention (and usable for up to five years). Teachers were to review student results on the school district's data analysis platform, then draft an action plan. Principals were asked to meet with teachers to review and revise the action plans and help develop re-teaching strategies for struggling students as well as teaching strategies for differentiated instruction. Compliance was tracked via submission of action plans. The third lever was observation and feedback. Principals were asked to observe teachers every other week for 15-20 minutes and later have a meeting to discuss takeaways. In addition, principals were asked to identify at least one action-step for the teacher to take to improve instruction. Compliance was monitored by tracking principal-teacher observations.

The researchers estimated both intent-to-treat (ITT) effects and local average treatment effects (LATEs) on principals' training attendance and training implementation and students' test scores. The researchers also examined heterogeneous effects of treatment.

Randomization occurred at the school-level. The sample frame was a list of 174 eligible schools (132 elementary, 22 middle, 19 high school) provided to the researchers by HISD. Fifty-eight schools from this list (20 elementary, 20 middle, 18 high school) made up the sample population. The researchers used matched-pair randomization to assign schools to treatment and control groups. First, sample schools were ranked by the sum of their mean scores on the high stakes state exams in math and reading for the previous two years. Next, every two schools going down the ranking were grouped, and one was randomly assigned to receive the treatment while the other served as the control.
Experimental Design Details
Randomization Method
At the beginning of the experiment, the school district gave the researchers a list of schools (separated into elementary, middle and high schools) eligible for the experiment. Schools were ranked by previous state test scores, and every two schools were designated a "matched pair". One school from each matched pair was randomly assigned to receive treatment, and the other served as control.
Randomization Unit
School-level randomization
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
58 schools (20 elementary schools, 20 middle schools, 18 high schools)
Sample size: planned number of observations
55,000 K-12th graders (25,000 treatment, 31,000 control) per year
Sample size (or number of clusters) by treatment arms
Treatment: 10 elementary schools, 10 middle schools, 9 high schools;
Control: 10 elementary schools, 10 middle schools, 9 high schools
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Supporting Documents and Materials

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
IRB

Institutional Review Boards (IRBs)

IRB Name
Harvard University
IRB Approval Date
2014-02-24
IRB Approval Number
14-0380

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
Yes
Intervention Completion Date
June 22, 2016, 12:00 +00:00
Data Collection Complete
Yes
Data Collection Completion Date
June 22, 2016, 12:00 +00:00
Final Sample Size: Number of Clusters (Unit of Randomization)
58 schools (20 elementary schools, 20 middle schools, 18 high schools)
Was attrition correlated with treatment status?
No
Final Sample Size: Total Number of Observations
55,000 K-12th graders (25,000 treatment, 31,000 control)
Final Sample Size (or Number of Clusters) by Treatment Arms
Treatment: 10 elementary schools, 10 middle schools, 9 high schools; Control: 10 elementary schools, 10 middle schools, 9 high schools
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
No
Reports, Papers & Other Materials

Relevant Paper(s)

Abstract
This study examines the impact on student achievement of implementing management training for principals in traditional public schools in Houston, Texas, using a school-level randomized field experiment. Across two years, principals were provided 300 hours of training on lesson planning, data-driven instruction, and teacher observation and coaching. The findings show that offering management training to principals significantly increases student achievement in all subjects in year one and has an insignificant effect in year two. We argue that the results in year two are driven by principal turnover, coupled with the cumulative nature of the training. Schools with principals who are predicted to remain in their positions for both years of the experiment demonstrate large treatment effects in both years - particularly those with principals who are also predicted to implement the training with high fidelity - while those with principals that are predicted to leave have statistically insignificant effects in each year of treatment.
Citation
Fryer, Roland G., Management and Student Achievement: Evidence from a Randomized Field Experiment (May 2017). NBER Working Paper No. w23437.
Abstract
This study examines the impact on student achievement of implementing management training for principals in traditional public schools in Houston, Texas, using a school-level randomized field experiment. Across two years, principals were provided 300 hours of training on lesson planning, data-driven instruction, and teacher observation and coaching. The findings show that offering management training to principals significantly increases student achievement in all subjects in year one and has an insignificant effect in year two. We argue that the results in year two are driven by principal turnover, coupled with the cumulative nature of the training. Schools with principals who are predicted to remain in their positions for both years of the experiment demonstrate large treatment effects in both years - particularly those with principals who are also predicted to implement the training with high fidelity - while those with principals that are predicted to leave have statistically insignificant effects in each year of treatment.
Citation
Fryer, Roland G., Management and Student Achievement: Evidence from a Randomized Field Experiment (May 2017). NBER Working Paper No. w23437.

Reports & Other Materials