The University of Chicago Education Lab and Crime Lab New York research teams are carrying out a randomized controlled trial during the 2016-17 and 2017-18 academic years to build on previous collaborations with the Chicago Public Schools (CPS), the New York City Department of Education, and Saga Education that have found that Saga's intensive, individualized, during-the-school-day tutoring can generate very large gains in academic outcomes in a short period of time. This research suggests the promise of this approach for improving the academic skills and educational attainment of disadvantaged youth, even once they have reached adolescence. However, to truly affect outcomes at the local and national level, Saga would have to be rolled out on a much greater scale than researchers have been able to study in Chicago. Yet little is known about how to take promising interventions to scale. As such, this study seeks to build the science of scale-up, by examining the extent to which this individualized tutoring program can be implemented at an even greater scale and by explicitly exploring the trade-offs between effectiveness and scale.
This study aims to build upon the investigators' previous evaluations of the program, and will provide insight into the ability of this program to serve youth at a much larger scale. Specifically, this study aims to answer the following research questions:
(1) What is the effect of implementing an evidence-based individualized tutoring program at larger scale?
(2) What is the relationship between the effect of the program and the scale at which the program is implemented?
Implementation sites are divided into two sets: sites in Chicago at which students are randomized to receive tutoring (hereby referred to as “scale-up” schools), and sites in Chicago and New York City where principals have primary discretion to choose which students receive tutoring (hereby referred to as “returning schools”). In addition to randomizing students into the Saga program, scale-up schools are also served by randomly selected tutors. The research team is having Saga over-recruit tutors as though they were implementing at larger than the intended scale in the scale-up schools. Investigators then randomly select one in three-and-a-half tutor applicants to continue through Saga’s standard hiring process, and positions at the scale-up schools are only filled by these randomly selected tutors. All study schools (both scale-up and returning) are implementing a third form of randomization: students in the Saga program are randomly assigned to tutors.
In order to study research question #1, investigators will take advantage of the power of random sampling of tutors and the random assignment into the Saga program in the scale-up schools to study scale up of this program without actually having to implement the program at a much larger scale. By comparing the outcomes of students randomly assigned to the Saga program to students randomly assigned to the control group in these schools, we will be able to rigorously estimate the effects of the program if it were being staffed by the tutors that would work for a program that is three-and-a-half times as large as the program currently operating in the scale-up schools.
To gain insight into research question #2 above, which seeks to determine the relationship between program scale and effects, tutors at all sites are ranked by Saga leadership based on relative expected quality. Because the research team is randomizing student pairs to tutors, we will be able to isolate the effect of value-add of each tutor in the Saga program. Combining this information with the Saga rankings of applicant quality, the researcher team will be able to examine tutor ranking and tutor effectiveness. Assuming that the program would hire tutors in the order of their ranking depending on the number of tutor slots they needed to fill, this analysis will shed light on the relationship between scale and effectiveness.
At the end of academic year (AY) 2016-17 and AY2017-18, researchers will answer the research questions by looking at various academic and behavioral outcomes accessed through administrative data from the Chicago Public Schools, New York City Department of Education, and Chicago Police Department. These outcomes include GPA, credits attempted, credits earned, and standardized test scores in Chicago and New York City, and crime involvement in Chicago. Researchers will also have access to data on tutor characteristics, which includes demographic information and information on math ability and teacher training, to assess the variation in the effectiveness of tutors and to examine the degree to which tutor characteristics correspond to different student outcomes. Lastly, researchers will receive programmatic attendance data from Saga, which includes information on how often students were present and which tutors they worked with each day. From this attendance data, we will be able to compute an “average tutor ranking”, which will be a weighted average of tutor ranking and how many days a student worked with a tutor of that ranking.
Researchers believe the proposed project will provide valuable insights into how to scale up what may turn out to be an extremely cost-effective way to improve educational outcomes for low-income youth. But perhaps more importantly, the hope is that this work will contribute to a broader understanding of how program scale affects program quality and how to systematically study this relationship. The investigators hope this project will create a "roadmap" for a growing body of research on the science of scale-up, guiding efforts of future researchers to understand the implications of scale-up for interventions with scarce inputs.