Increasing the reach of promising dropout prevention programs: Examining the tradeoffs between scale and effectiveness

SAGA Innovations’ tutoring model provides youth with high-intensity, individualized math tutoring—two-on-one instruction for an hour every day, during the school day—designed to help youth catch back up to grade level so that they can re-engage with regular classroom instruction. The SAGA Innovations program expands on the nationally recognized innovation of high-dosage, in-school-day tutoring developed in Match Education's charter school in Boston. The tutoring program meets as a scheduled course, Math Lab, once a day during the normal school day, and is provided in addition to a student's regular math class. Students work two-on-one (two students with one tutor) with the same full-time, professional tutor for the entirety of the school year. The content of the tutoring sessions is aligned with what students are learning in their regular math courses, but is also targeted to address individual gaps in math knowledge. Also following the original model developed by Match Education, SAGA tutors use frequent internal formative assessments of student progress to individualize instruction.

A previous randomized controlled trial conducted by the University of Chicago research team found that one year of this intervention, delivered in AY2013-14 in the Chicago Public Schools, generated between one and two extra years of academic growth in math, over and above what the normal U.S. high school student learns in one year (Cook et al., 2015; Reardon, 2011). The estimated effects for math achievement are on the order of 0.19 to 0.30 standard deviations, depending on the exact test and norming used. The intervention also improved student grades in math by 0.58 points on a 1-4 grade point scale, compared to a control mean of 1.77. These gains are particularly important because of the growing evidence on the importance of math specifically for short- and medium-term success in school, and for long-term life outcomes such as employment and earnings (Duncan et al., 2007).

2016-09-01

2018-06-30

Difference in math achievement (end-of-year)

Performance on end-of-year math standardized achievement tests, obtained from Chicago Public Schools and New York City Department of Education administrative data. The outcome will be standardized scaled scores on these tests, and our primary outcome will be the treatment-on-the-treated effect. Our scores are standardized based on the mean and standard deviation of the control group for each grade. Using these means and standard deviations, raw scores and percentiles are transformed into z-scores so that one can interpret the results as the standard deviation change from the control group in each grade.

Secondary outcomes include: difference in math achievement (two and three years post-intervention), difference in math course grades, non-math course grades, standardized test score achievement on non-math exam sections, absentee rate, index of schooling outcomes (e.g. Z-score of outcomes for school persistence, absences, and course grades), school misconduct, total courses failed, math courses failed, non-math courses failed, school persistence, violent crime arrests (research team is currently unsure if we can obtain this information for NYC), other arrests (property/drug/other crimes; research team is currently unsure if we can obtain this information for NYC). All of these outcomes will be measured at end-of-year, as well as two and three years post-intervention. We will also be including outcomes such as high-school graduation (time frame: 3 years, 4 years, and 5 years after 9th and 10th grade enrollment), as well as college enrollment rate (time frame: 3 years, 4 years, 5 years, 6 years, 7 years, and 8 years post-high school enrollment).

All of our academic outcome data will be collected through administrative data from the Chicago Public Schools and New York City Department of Education. Arrest outcome data will be collected through administrative data from the Chicago Police Department and Illinois State Police. The research team will additionally attempt to obtain arrest data from the New York City Police Department. More information on our outcomes, and how they will be calculated can be found in attachments in the "Analysis Plan" section.

The University of Chicago Education Lab and Crime Lab New York research teams are carrying out a randomized controlled trial during the 2016-17 and 2017-18 academic years to build on previous collaborations with the Chicago Public Schools (CPS), the New York City Department of Education, and SAGA Innovations that have found that SAGA's intensive, individualized, during-the-school-day tutoring can generate very large gains in academic outcomes in a short period of time. This research suggests the promise of this approach for improving the academic skills and educational attainment of disadvantaged youth, even once they have reached adolescence. However, to truly affect outcomes at the local and national level, SAGA would have to be rolled out on a much greater scale than researchers have been able to study in Chicago. Yet little is known about how to take promising interventions to scale. As such, this study seeks to build the science of scale-up, by examining the extent to which this individualized tutoring program can be implemented at an even greater scale and by explicitly exploring the trade-offs between effectiveness and scale.

This study aims to build upon the investigators' previous evaluations of the program, and will provide insight into the ability of this program to serve youth at a much larger scale. Specifically, this study aims to answer the following research questions:
(1) What is the effect of implementing an evidence-based individualized tutoring program at larger scale?
(2) What is the relationship between the effect of the program and the scale at which the program is implemented?

Implementation sites are divided into two sets: sites in Chicago at which students are randomized to receive tutoring (hereby referred to as “scale-up” schools), and sites in Chicago and New York City where principals have primary discretion to choose which students receive tutoring (hereby referred to as “returning schools”). In addition to randomizing students into the SAGA program, scale-up schools are also served by randomly selected tutors. The research team is having SAGA over-recruit tutors as though they were implementing at larger than the intended scale in the scale-up schools. Investigators then randomly select one in three-and-a-half tutor applicants to continue through SAGA’s standard hiring process, and positions at the scale-up schools are only filled by these randomly selected tutors. All study schools (both scale-up and returning) are implementing a third form of randomization: students in the SAGA program are randomly assigned to tutors.

In order to study research question #1, investigators will take advantage of the power of random sampling of tutors and the random assignment into the SAGA program in the scale-up schools to study scale up of this program without actually having to implement the program at a much larger scale. By comparing the outcomes of students randomly assigned to the SAGA program to students randomly assigned to the control group in these schools, we will be able to rigorously estimate the effects of the program if it were being staffed by the tutors that would work for a program that is three-and-a-half times as large as the program currently operating in the scale-up schools.

To gain insight into research question #2 above, which seeks to determine the relationship between program scale and effects, tutors at all sites are ranked by SAGA leadership based on relative expected quality. Because the research team is randomizing student pairs to tutors, we will be able to isolate the effect of value-add of each tutor in the SAGA program. Combining this information with the SAGA rankings of applicant quality, the researcher team will be able to examine tutor ranking and tutor effectiveness. Assuming that the program would hire tutors in the order of their ranking depending on the number of tutor slots they needed to fill, this analysis will shed light on the relationship between scale and effectiveness.

At the end of academic year (AY) 2016-17 and AY2017-18, researchers will answer the research questions by looking at various academic and behavioral outcomes accessed through administrative data from the Chicago Public Schools, New York City Department of Education, and Chicago Police Department. These outcomes include GPA, credits attempted, credits earned, and standardized test scores in Chicago and New York City, and crime involvement in Chicago. Researchers will also have access to data on tutor characteristics, which includes demographic information and information on math ability and teacher training, to assess the variation in the effectiveness of tutors and to examine the degree to which tutor characteristics correspond to different student outcomes. Lastly, researchers will receive programmatic attendance data from SAGA, which includes information on how often students were present and which tutors they worked with each day. From this attendance data, we will be able to compute an “average tutor ranking”, which will be a weighted average of tutor ranking and how many days a student worked with a tutor of that ranking.

Researchers believe the proposed project will provide valuable insights into how to scale up what may turn out to be an extremely cost-effective way to improve educational outcomes for low-income youth. But perhaps more importantly, the hope is that this work will contribute to a broader understanding of how program scale affects program quality and how to systematically study this relationship. The investigators hope this project will create a "roadmap" for a growing body of research on the science of scale-up, guiding efforts of future researchers to understand the implications of scale-up for interventions with scarce inputs.

There are three levels of randomization into our study: tutor applicant randomization at a subset of the Chicago schools; treatment assignment randomization at a subset of the Chicago schools; and student pair-to-tutor randomization that occurs at all schools across Chicago and NYC. All three randomizations will occur in an office by a computer. More specifically, the tutor applicant randomization will be done via the "random number" function Excel; the treatment assignment and student-tutor randomization will be done via Stata.

As noted above, there are three levels of randomization into our study: tutor applicant randomization at a subset of the Chicago schools; treatment assignment randomization at a subset of the Chicago schools; and student pair-tutor randomization that occurs at all schools across Chicago and NYC. Tutor applicant randomization occurs at the individual-level. Treatment assignment randomization is blocked by school, grade, and gender, and occurs at the individual level. Student pair-tutor randomization occurs at the student pair level.

No

In academic year 2016-17, we have 11 "sites" participating in the study (or 15 total schools, as some schools are combined into one "site") in Chicago. We also have 3 sites in NYC (or 5 total schools).

In academic year 2017-18, we have 11 total implementation sites in Chicago (and 16 total high schools), and 3 total implementation sites in NYC (and 4 total schools).

In 2016-17, we expect about 2,700 students and 100 tutors to participate in this study in Chicago, and 700 students and 51 tutors in NYC. In 2017-18, we expect about 2,400 students and 100 tutors to participate in this study in Chicago, and 700 students and 45 tutors in NYC. In total, we will have about 6,800 observations.

Of the students who are randomized in Chicago, we expect approximately 1,200 will be randomized to control and 2,500 will be randomized to treatment. Of the tutor applicants we randomize in Chicago, approximately one in every three-and-a-half will be randomized to continue along the hiring process. All students in our study (about 6,500) will be randomized to a tutor (approximately 300 total).

Analysis Plan - Revision #1

Analysis Plan