Evaluation of the District of Columbia Public Schools Family Engagement Partnership

Last registered on June 23, 2016

Pre-Trial

Trial Information

General Information

Title
Evaluation of the District of Columbia Public Schools Family Engagement Partnership
RCT ID
AEARCTR-0001370
Initial registration date
June 23, 2016

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
June 23, 2016, 9:58 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
June 23, 2016, 10:00 AM EDT

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Primary Investigator

Affiliation
Mathematica Policy Research

Other Primary Investigator(s)

PI Affiliation
Mathematica Policy Research

Additional Trial Information

Status
Withdrawn
Start date
2014-01-15
End date
2015-12-18
Secondary IDs
Abstract
The District of Columbia Public Schools (DCPS) partnered with the Flamboyan Foundation to implement the Family Engagement Partnership (FEP), a two-year school-based intervention that aims to increase family engagement and, ultimately, student achievement. As part of this effort, DCPS selected Mathematica Policy Research, a nonpartisan research firm, to serve as the independent evaluator of FEP. Mathematica conducted a randomized controlled trial with a multi-step screening process to estimate the impacts of FEP on family engagement, student achievement, and other key outcomes. The treatment in this intent-to-treat evaluation was the invitation to apply to FEP, not necessarily the implementation of FEP. In addition to using administrative data to measure student and teacher outcomes, we planned to develop and administer surveys to assess family engagement activities and outcomes. Teacher surveys conducted in spring 2015, spring 2016, and spring 2018 would allow us to examine whether FEP activities differ from usual practices in control schools and estimate impacts on teacher satisfaction and perceptions of student-teacher relationships. We planned to conduct a parent survey in summer 2016 and summer 2018 to measure family engagement outcomes from the parents’ perspective. Findings were to be presented in a 2019 report. However, the study sponsors decided to end the evaluation early due to statistical power considerations. Preliminary findings from this evaluation and another study’s recent findings suggested that impacts of the intervention may be difficult to detect in part because they may be modest in size and not statistically significant with the sample of schools participating in the study.
External Link(s)

Registration Citation

Citation
Glazerman, Steven and Allison McKie. 2016. "Evaluation of the District of Columbia Public Schools Family Engagement Partnership." AEA RCT Registry. June 23. https://doi.org/10.1257/rct.1370-2.0
Former Citation
Glazerman, Steven and Allison McKie. 2016. "Evaluation of the District of Columbia Public Schools Family Engagement Partnership." AEA RCT Registry. June 23. https://www.socialscienceregistry.org/trials/1370/history/8970
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
FEP is a two-year, school-based intervention that aims to increase family engagement and, ultimately, student achievement, via activities such as home visits, academic parent-teacher teams, and ongoing teacher communication with families.
Intervention Start Date
2014-04-30
Intervention End Date
2015-12-18

Primary Outcomes

Primary Outcomes (end points)
The main outcomes are students’ attachment to school, student achievement, teacher retention, teacher performance, and family engagement.
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
Mathematica designed a randomized controlled trial with a multi-step screening process. In Step 1, DCPS screened schools based on extant information for suitability in FEP. This step did not involve outreach to candidate schools themselves. Step 2 involved minimal outreach to schools of marginal suitability to gauge interest and capacity. The outreach was not rigorous enough to raise schools’ hopes or change their behavior. Schools passing this screen, however, had to consent to participate in the study. At the conclusion of this step, DCPS provided Mathematica with a list of schools to randomly assign.

In Step 3, the Mathematica evaluation team randomly assigned half of the schools to a treatment group and half to a control group. Mathematica first used school-level data to create pairs of schools that were similar on observable characteristics, such as average student achievement and student demographics. We then consulted with DCPS to ensure that district staff and program implementers who were familiar with the schools deemed each proposed pair of schools to be similar. Once the matched pairs were agreed upon, we randomly assigned one school in each pair to be invited to apply for FEP (treatment group); the other school was not invited to apply (control group).

FEP staff then rigorously screened the treatment schools and worked with the treatment schools that were selected for the intervention. The evaluation team planned to follow all study schools over time. In this intent-to-treat evaluation, schools assigned to treatment remain in the treatment group for the study, regardless of whether they implemented FEP.
Experimental Design Details
The original evaluation plan called for analyzing one cohort of 40 schools (20 treatment and 20 control sites). However, only 18 schools passed the initial FEP screening and entered the randomization pool in January 2014. Nine treatment schools were invited to apply to implement FEP for the 2014–2015 and 2015–2016 school years. The 18 randomly assigned schools by treatment status were as follows:
• Treatment schools: Burroughs Education Campus (EC), Burrville, J.O. Wilson, Ketcham, Kimball, Langley, Maury, Savoy, and Takoma EC.
• Control schools: Browne EC, Cleveland, Drew, J.C. Nalle, Ludlow-Taylor, Orr, Randle Highlands, Raymond EC, and Whittier EC.

The evaluation plan was modified in March 2015 to add a second cohort of 22 schools to be randomly assigned in December 2015. The treatment schools would be invited to apply to implement FEP for the 2016–2017 and 2017–2018 school years. However, the evaluation was terminated prior to assigning the second cohort of schools.
Randomization Method
Mathematica conducted randomization within matched pairs of schools using a random number generator.
Randomization Unit
School.
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
40 schools.
Sample size: planned number of observations
Approximately 10,200 students with non-missing administrative data, 400 teachers in the teacher survey sample, and 400 parents in the parent survey sample.
Sample size (or number of clusters) by treatment arms
As planned: 20 treatment schools and 20 control schools.
As implemented: 9 treatment schools and 9 control schools.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
IRB

Institutional Review Boards (IRBs)

IRB Name
Health Media Lab
IRB Approval Date
2015-05-22
IRB Approval Number
Details not available

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials