Improving and Sustaining Management Practices in Public Schools: Evidence from Puerto Rico
Last registered on May 29, 2019

Pre-Trial

Trial Information
General Information
Title
Improving and Sustaining Management Practices in Public Schools: Evidence from Puerto Rico
RCT ID
AEARCTR-0004066
Initial registration date
April 09, 2019
Last updated
May 29, 2019 9:32 AM EDT
Location(s)

This section is unavailable to the public. Use the button below to request access to this information.

Request Information
Primary Investigator
Affiliation
UC-Berkeley
Other Primary Investigator(s)
PI Affiliation
University of Toronto
PI Affiliation
Cornell University
Additional Trial Information
Status
In development
Start date
2018-08-13
End date
2021-07-21
Secondary IDs
Abstract
Improvements in management practices have been credited with generating stunning productivity improvement in the private sector. Although their adoption in the education sector has been slow, management-level interventions have been shown to lead to significant improvements in school quality and students’ academic achievement. However, adoption of these practices in the public sector is often not sustained in the long run. Promoting improvements in school management practices is thus an amenable area for policy intervention, given the need to identify ways to achieve this sustainably and at scale.

Our proposed study is a collaboration with the Puerto Rico Department of Education (PRDE) to experimentally evaluate the impact of a large-scale principal training program among its school directors. We will measure impacts on managerial practices collecting detailed longitudinal data to measure short and long-term adoption, as well as impacts on student achievement using standardized test scores data. The study will help inform whether improvements in managerial practices can be implemented in schools at scale, their long-term sustainability, and their effects on student learning.
External Link(s)
Registration Citation
Citation
Bobonis, Gustavo, Marco Gonzalez-Navarro and Daniela Scur. 2019. "Improving and Sustaining Management Practices in Public Schools: Evidence from Puerto Rico." AEA RCT Registry. May 29. https://www.socialscienceregistry.org/trials/4066/history/47256
Sponsors & Partners

There are documents in this trial unavailable to the public. Use the button below to request access to this information.

Request Information
Experimental Details
Interventions
Intervention(s)
As part of the government’s policy priorities, the Puerto Rico Department of Education (PRDE) is seeking to introduce school leadership quality reforms to improve educational outcomes in the territory. The study will rigorously evaluate – via a randomized controlled trial – the impact of EDUGESPRO; a large-scale management training program to be implemented among the universe of approximately 900 PRDE school principals in the territory’s public education system over a two-year period (Academic Years 2010-20, 2020-21). The training of each cohort of principals will take place over the course of one academic year (approximately 154 hours), composed of an intensive 2-week summer intensive session in addition to monthly workshops and follow-up support during the academic year. A second treatment arm will also receive a post-training intervention aimed at sustaining managerial upgrades and maintenance of learned strategies via reminders.
Intervention Start Date
2019-06-03
Intervention End Date
2021-05-31
Primary Outcomes
Primary Outcomes (end points)
student standardized test scores
Primary Outcomes (explanation)
Secondary Outcomes
Secondary Outcomes (end points)
school-level managerial practices index; principal attendance; teacher attendance; classroom instruction; classroom management; student attendance; student grades
Secondary Outcomes (explanation)
The school-level managerial practices index is constructed using the Development World Management Survey (D-WMS), an internationally validated tool that helps identify prevalent management practices in schools. Classroom-level instruction and management measures will be captured using standard classroom observation instruments used in national and international settings (e.g. TEACH, Stallings, CLASS). Principal, teacher, and student attendance, as well as student grades, are based on PRDE administrative data.
Experimental Design
Experimental Design
The proposed design of the Puerto Rico Principals’ Academy program comprises two stages. Approximately 819 PRDE schools and their principals will be stratified by academic level (i.e., K-8, 9-12), PRDE district/region, and adoption of modern management practices at baseline. All schools within each stratum will be randomly assigned to one of three (3) experimental arms:

Treatment Arm 1 – Intensive Management Training (IMT) (Summer 2019, AY 2019-20) + Adoption Reminders (AYs 2019-20, 2020-21);
Treatment Arm 2 – IMT (Summer 2020, AY 2020-21) + Adoption Reminders (AY 2020-21);
Control Arm – Status Quo (Placebo)

Each of the arms of this stepped-wedge design will be composed of 273 schools/principals, with principals in all schools receiving the IMT by the end of the third year.
Experimental Design Details
Not available
Randomization Method
Stratified randomization by academic level (i.e., K-8, 9-12), PRDE district/region, and predicted adoption of modern management practices at baseline, done in office by a computer.
Randomization Unit
School
Was the treatment clustered?
Yes
Experiment Characteristics
Sample size: planned number of clusters
817 schools
Sample size: planned number of observations
140,000 students
Sample size (or number of clusters) by treatment arms
Treatment Arm 1 – 273 schools
Treatment Arm 2 – 272 schools
Control Arm - 272 schools
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Power calculations for students’ subject-specific proficiency rates based on standardized test scores, by academic level (i.e., Grades 3-8, Grade 11), using school-level administrative data for AY 2015-16 & 2016-17. Considering the cluster-randomized design with measurements of student test scores at baseline and end-line, the power calculations (power=0.9, α=0.05) suggest that with 237 elementary/middle schools per arm (assuming equal treatment and control group sample sizes), it will be possible to detect a year-specific increase in Grades 3-8 students’ academic proficiency in Spanish, English, and Math scores of respectively 0.11s, 0.10s, and 0.09s. (See power analysis appendix and response for details.) This is similar to existing short-term impacts of principal training programs in the US using student-level outcomes (e.g., Fryer 2017, Gates et al. 2014). Comparable MDEs for Grade 11 test scores are somewhat larger –in the range of 0.27s (in English) to 0.31s (in Spanish) – given the more limited number of high schools (61 high schools per experimental arm). We expect to gain statistical power at both grade level groups since we will have access to individual students’ (continuous) standardized test scores as opposed to only (binary) proficiency levels, individual-level baseline outcomes, and a larger sample of schools in the control group in Year 1 [e.g., 474 (= 273 x 2) K-8 schools and 122 (= 61 x 2) high schools].
IRB
INSTITUTIONAL REVIEW BOARDS (IRBs)
IRB Name
University of Toronto
IRB Approval Date
2018-03-18
IRB Approval Number
35656
Analysis Plan

There are documents in this trial unavailable to the public. Use the button below to request access to this information.

Request Information