Improving and Sustaining Management Practices in Public Schools: Evidence from Puerto Rico

Last registered on May 29, 2019

Pre-Trial

Trial Information

General Information

Title
Improving and Sustaining Management Practices in Public Schools: Evidence from Puerto Rico
RCT ID
AEARCTR-0004066
Initial registration date
April 09, 2019

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
April 17, 2019, 8:21 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
May 29, 2019, 9:32 AM EDT

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Region

Primary Investigator

Affiliation
UC-Berkeley

Other Primary Investigator(s)

PI Affiliation
University of Toronto
PI Affiliation
Cornell University

Additional Trial Information

Status
In development
Start date
2018-08-13
End date
2021-07-21
Secondary IDs
Abstract
Improvements in management practices have been credited with generating stunning productivity improvement in the private sector. Although their adoption in the education sector has been slow, management-level interventions have been shown to lead to significant improvements in school quality and students’ academic achievement. However, adoption of these practices in the public sector is often not sustained in the long run. Promoting improvements in school management practices is thus an amenable area for policy intervention, given the need to identify ways to achieve this sustainably and at scale.

Our proposed study is a collaboration with the Puerto Rico Department of Education (PRDE) to experimentally evaluate the impact of a large-scale principal training program among its school directors. We will measure impacts on managerial practices collecting detailed longitudinal data to measure short and long-term adoption, as well as impacts on student achievement using standardized test scores data. The study will help inform whether improvements in managerial practices can be implemented in schools at scale, their long-term sustainability, and their effects on student learning.
External Link(s)

Registration Citation

Citation
Bobonis, Gustavo, Marco Gonzalez-Navarro and Daniela Scur. 2019. "Improving and Sustaining Management Practices in Public Schools: Evidence from Puerto Rico." AEA RCT Registry. May 29. https://doi.org/10.1257/rct.4066-2.0
Former Citation
Bobonis, Gustavo, Marco Gonzalez-Navarro and Daniela Scur. 2019. "Improving and Sustaining Management Practices in Public Schools: Evidence from Puerto Rico." AEA RCT Registry. May 29. https://www.socialscienceregistry.org/trials/4066/history/47256
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
As part of the government’s policy priorities, the Puerto Rico Department of Education (PRDE) is seeking to introduce school leadership quality reforms to improve educational outcomes in the territory. The study will rigorously evaluate – via a randomized controlled trial – the impact of EDUGESPRO; a large-scale management training program to be implemented among the universe of approximately 900 PRDE school principals in the territory’s public education system over a two-year period (Academic Years 2010-20, 2020-21). The training of each cohort of principals will take place over the course of one academic year (approximately 154 hours), composed of an intensive 2-week summer intensive session in addition to monthly workshops and follow-up support during the academic year. A second treatment arm will also receive a post-training intervention aimed at sustaining managerial upgrades and maintenance of learned strategies via reminders.
Intervention (Hidden)
Intensive Management Training: Two-week (70 hours) Intensive Management Training Sessions (Summer) + Monthly Workshops (68 hours) during academic year + Personalized Coaching (16 hours). The curriculum will be developed jointly by the PRDE and the training provider organization, as well as focus areas informed by the research team’s analysis of the baseline adoption of management practices: operations (instructional leadership), monitoring, target setting, and personnel management. The workshops to be carried out throughout each academic year will help reinforce concepts, address challenges, and make actionable adjustments in schools.

Adoption Reminders: Principals will also receive targeted messages reminding them of the management practices and tailored tips (based on the school management baseline data) to implement these; others will receive a placebo message. Depending on the connectivity at the time of implementation, we will experiment with individualized text messages or group text messages (allowing also for a measurement of network formation and practice dissemination).

Placebo intervention: School principals from all three arms will receive a placebo treatment: a set of administrative and basic management training sessions to be carried out during AY 2018-19. These basic training sessions will be focused on compliance with PRDE information and regulatory requirements as well as basic administrative skills.
Intervention Start Date
2019-06-03
Intervention End Date
2021-05-31

Primary Outcomes

Primary Outcomes (end points)
student standardized test scores
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
school-level managerial practices index; principal attendance; teacher attendance; classroom instruction; classroom management; student attendance; student grades
Secondary Outcomes (explanation)
The school-level managerial practices index is constructed using the Development World Management Survey (D-WMS), an internationally validated tool that helps identify prevalent management practices in schools. Classroom-level instruction and management measures will be captured using standard classroom observation instruments used in national and international settings (e.g. TEACH, Stallings, CLASS). Principal, teacher, and student attendance, as well as student grades, are based on PRDE administrative data.

Experimental Design

Experimental Design
The proposed design of the Puerto Rico Principals’ Academy program comprises two stages. Approximately 819 PRDE schools and their principals will be stratified by academic level (i.e., K-8, 9-12), PRDE district/region, and adoption of modern management practices at baseline. All schools within each stratum will be randomly assigned to one of three (3) experimental arms:

Treatment Arm 1 – Intensive Management Training (IMT) (Summer 2019, AY 2019-20) + Adoption Reminders (AYs 2019-20, 2020-21);
Treatment Arm 2 – IMT (Summer 2020, AY 2020-21) + Adoption Reminders (AY 2020-21);
Control Arm – Status Quo (Placebo)

Each of the arms of this stepped-wedge design will be composed of 273 schools/principals, with principals in all schools receiving the IMT by the end of the third year.
Experimental Design Details
Randomization Method
Stratified randomization by academic level (i.e., K-8, 9-12), PRDE district/region, and predicted adoption of modern management practices at baseline, done in office by a computer.
Randomization Unit
School
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
817 schools
Sample size: planned number of observations
140,000 students
Sample size (or number of clusters) by treatment arms
Treatment Arm 1 – 273 schools
Treatment Arm 2 – 272 schools
Control Arm - 272 schools
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Power calculations for students’ subject-specific proficiency rates based on standardized test scores, by academic level (i.e., Grades 3-8, Grade 11), using school-level administrative data for AY 2015-16 & 2016-17. Considering the cluster-randomized design with measurements of student test scores at baseline and end-line, the power calculations (power=0.9, α=0.05) suggest that with 237 elementary/middle schools per arm (assuming equal treatment and control group sample sizes), it will be possible to detect a year-specific increase in Grades 3-8 students’ academic proficiency in Spanish, English, and Math scores of respectively 0.11s, 0.10s, and 0.09s. (See power analysis appendix and response for details.) This is similar to existing short-term impacts of principal training programs in the US using student-level outcomes (e.g., Fryer 2017, Gates et al. 2014). Comparable MDEs for Grade 11 test scores are somewhat larger –in the range of 0.27s (in English) to 0.31s (in Spanish) – given the more limited number of high schools (61 high schools per experimental arm). We expect to gain statistical power at both grade level groups since we will have access to individual students’ (continuous) standardized test scores as opposed to only (binary) proficiency levels, individual-level baseline outcomes, and a larger sample of schools in the control group in Year 1 [e.g., 474 (= 273 x 2) K-8 schools and 122 (= 61 x 2) high schools].
IRB

Institutional Review Boards (IRBs)

IRB Name
University of Toronto
IRB Approval Date
2018-03-18
IRB Approval Number
35656
Analysis Plan

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials