|
Field
Abstract
|
Before
The inability to consistently deliver promising education interventions at large scale is an important contributing cause to inequality in the U.S. The research team applies insights from price theory and field-based randomized controlled trials to examine the effect of implementing a promising academic skills development program at large scale before implementing at scale. The project is designed to provide evidence of direct scientific and policy value for attempts to scale-up a specific intervention, but also stimulate much more investigation of social policy scale-up challenges by refining these methods and demonstrating their feasibility and value.
The research team examines the challenge of program scale-up for a promising intervention studied in Chicago at medium scale in the past - "SAGA tutoring". Past work has demonstrated that SAGA's intensive, individualized, during-the-school-day math tutoring can generate very large gains in academic outcomes in a short period, even among students who are many years behind grade level. This study will explicitly explore the extent to which there is a trade-off between effectiveness and scale for this intervention. By taking advantage of the power of random sampling, this study will also allow for observation of the program's effectiveness as if it were running at three-and-a-half times the proposed scale in a subset of the study population.
|
After
The inability to consistently deliver promising education interventions at large scale is an important contributing cause to inequality in the U.S. The research team applies insights from price theory and field-based randomized controlled trials to examine the effect of implementing a promising academic skills development program at large scale before implementing at scale. The project is designed to provide evidence of direct scientific and policy value for attempts to scale-up a specific intervention, but also stimulate much more investigation of social policy scale-up challenges by refining these methods and demonstrating their feasibility and value.
The research team examines the challenge of program scale-up for a promising intervention studied in Chicago at medium scale in the past - "Saga tutoring". Past work has demonstrated that Saga's intensive, individualized, during-the-school-day math tutoring can generate very large gains in academic outcomes in a short period, even among students who are many years behind grade level. This study will explicitly explore the extent to which there is a trade-off between effectiveness and scale for this intervention. By taking advantage of the power of random sampling, this study will also allow for observation of the program's effectiveness as if it were running at three-and-a-half times the proposed scale in a subset of the study population.
|
|
Field
Intervention (Public)
|
Before
SAGA Innovations’ tutoring model provides youth with high-intensity, individualized math tutoring—two-on-one instruction for an hour every day, during the school day—designed to help youth catch back up to grade level so that they can re-engage with regular classroom instruction. The SAGA Innovations program expands on the nationally recognized innovation of high-dosage, in-school-day tutoring developed in Match Education's charter school in Boston. The tutoring program meets as a scheduled course, Math Lab, once a day during the normal school day, and is provided in addition to a student's regular math class. Students work two-on-one (two students with one tutor) with the same full-time, professional tutor for the entirety of the school year. The content of the tutoring sessions is aligned with what students are learning in their regular math courses, but is also targeted to address individual gaps in math knowledge. Also following the original model developed by Match Education, SAGA tutors use frequent internal formative assessments of student progress to individualize instruction.
A previous randomized controlled trial conducted by the University of Chicago research team found that one year of this intervention, delivered in AY2013-14 in the Chicago Public Schools, generated between one and two extra years of academic growth in math, over and above what the normal U.S. high school student learns in one year (Cook et al., 2015; Reardon, 2011). The estimated effects for math achievement are on the order of 0.19 to 0.30 standard deviations, depending on the exact test and norming used. The intervention also improved student grades in math by 0.58 points on a 1-4 grade point scale, compared to a control mean of 1.77. These gains are particularly important because of the growing evidence on the importance of math specifically for short- and medium-term success in school, and for long-term life outcomes such as employment and earnings (Duncan et al., 2007).
|
After
Saga Education’s tutoring model provides youth with high-intensity, individualized math tutoring—two-on-one instruction for an hour every day, during the school day—designed to help youth catch back up to grade level so that they can re-engage with regular classroom instruction. The Saga Education program expands on the nationally recognized innovation of high-dosage, in-school-day tutoring developed in Match Education's charter school in Boston. The tutoring program meets as a scheduled course, Math Lab, once a day during the normal school day, and is provided in addition to a student's regular math class. Students work two-on-one (two students with one tutor) with the same full-time, professional tutor for the entirety of the school year. The content of the tutoring sessions is aligned with what students are learning in their regular math courses, but is also targeted to address individual gaps in math knowledge. Also following the original model developed by Match Education, Saga tutors use frequent internal formative assessments of student progress to individualize instruction.
A previous randomized controlled trial conducted by the University of Chicago research team found that one year of this intervention, delivered in AY2013-14 in the Chicago Public Schools, generated between one and two extra years of academic growth in math, over and above what the normal U.S. high school student learns in one year (Cook et al., 2015; Reardon, 2011). The estimated effects for math achievement are on the order of 0.19 to 0.30 standard deviations, depending on the exact test and norming used. The intervention also improved student grades in math by 0.58 points on a 1-4 grade point scale, compared to a control mean of 1.77. These gains are particularly important because of the growing evidence on the importance of math specifically for short- and medium-term success in school, and for long-term life outcomes such as employment and earnings (Duncan et al., 2007).
|
|
Field
Experimental Design (Public)
|
Before
The University of Chicago Education Lab and Crime Lab New York research teams are carrying out a randomized controlled trial during the 2016-17 and 2017-18 academic years to build on previous collaborations with the Chicago Public Schools (CPS), the New York City Department of Education, and SAGA Innovations that have found that SAGA's intensive, individualized, during-the-school-day tutoring can generate very large gains in academic outcomes in a short period of time. This research suggests the promise of this approach for improving the academic skills and educational attainment of disadvantaged youth, even once they have reached adolescence. However, to truly affect outcomes at the local and national level, SAGA would have to be rolled out on a much greater scale than researchers have been able to study in Chicago. Yet little is known about how to take promising interventions to scale. As such, this study seeks to build the science of scale-up, by examining the extent to which this individualized tutoring program can be implemented at an even greater scale and by explicitly exploring the trade-offs between effectiveness and scale.
This study aims to build upon the investigators' previous evaluations of the program, and will provide insight into the ability of this program to serve youth at a much larger scale. Specifically, this study aims to answer the following research questions:
(1) What is the effect of implementing an evidence-based individualized tutoring program at larger scale?
(2) What is the relationship between the effect of the program and the scale at which the program is implemented?
Implementation sites are divided into two sets: sites in Chicago at which students are randomized to receive tutoring (hereby referred to as “scale-up” schools), and sites in Chicago and New York City where principals have primary discretion to choose which students receive tutoring (hereby referred to as “returning schools”). In addition to randomizing students into the SAGA program, scale-up schools are also served by randomly selected tutors. The research team is having SAGA over-recruit tutors as though they were implementing at larger than the intended scale in the scale-up schools. Investigators then randomly select one in three-and-a-half tutor applicants to continue through SAGA’s standard hiring process, and positions at the scale-up schools are only filled by these randomly selected tutors. All study schools (both scale-up and returning) are implementing a third form of randomization: students in the SAGA program are randomly assigned to tutors.
In order to study research question #1, investigators will take advantage of the power of random sampling of tutors and the random assignment into the SAGA program in the scale-up schools to study scale up of this program without actually having to implement the program at a much larger scale. By comparing the outcomes of students randomly assigned to the SAGA program to students randomly assigned to the control group in these schools, we will be able to rigorously estimate the effects of the program if it were being staffed by the tutors that would work for a program that is three-and-a-half times as large as the program currently operating in the scale-up schools.
To gain insight into research question #2 above, which seeks to determine the relationship between program scale and effects, tutors at all sites are ranked by SAGA leadership based on relative expected quality. Because the research team is randomizing student pairs to tutors, we will be able to isolate the effect of value-add of each tutor in the SAGA program. Combining this information with the SAGA rankings of applicant quality, the researcher team will be able to examine tutor ranking and tutor effectiveness. Assuming that the program would hire tutors in the order of their ranking depending on the number of tutor slots they needed to fill, this analysis will shed light on the relationship between scale and effectiveness.
At the end of academic year (AY) 2016-17 and AY2017-18, researchers will answer the research questions by looking at various academic and behavioral outcomes accessed through administrative data from the Chicago Public Schools, New York City Department of Education, and Chicago Police Department. These outcomes include GPA, credits attempted, credits earned, and standardized test scores in Chicago and New York City, and crime involvement in Chicago. Researchers will also have access to data on tutor characteristics, which includes demographic information and information on math ability and teacher training, to assess the variation in the effectiveness of tutors and to examine the degree to which tutor characteristics correspond to different student outcomes. Lastly, researchers will receive programmatic attendance data from SAGA, which includes information on how often students were present and which tutors they worked with each day. From this attendance data, we will be able to compute an “average tutor ranking”, which will be a weighted average of tutor ranking and how many days a student worked with a tutor of that ranking.
Researchers believe the proposed project will provide valuable insights into how to scale up what may turn out to be an extremely cost-effective way to improve educational outcomes for low-income youth. But perhaps more importantly, the hope is that this work will contribute to a broader understanding of how program scale affects program quality and how to systematically study this relationship. The investigators hope this project will create a "roadmap" for a growing body of research on the science of scale-up, guiding efforts of future researchers to understand the implications of scale-up for interventions with scarce inputs.
|
After
The University of Chicago Education Lab and Crime Lab New York research teams are carrying out a randomized controlled trial during the 2016-17 and 2017-18 academic years to build on previous collaborations with the Chicago Public Schools (CPS), the New York City Department of Education, and Saga Education that have found that Saga's intensive, individualized, during-the-school-day tutoring can generate very large gains in academic outcomes in a short period of time. This research suggests the promise of this approach for improving the academic skills and educational attainment of disadvantaged youth, even once they have reached adolescence. However, to truly affect outcomes at the local and national level, Saga would have to be rolled out on a much greater scale than researchers have been able to study in Chicago. Yet little is known about how to take promising interventions to scale. As such, this study seeks to build the science of scale-up, by examining the extent to which this individualized tutoring program can be implemented at an even greater scale and by explicitly exploring the trade-offs between effectiveness and scale.
This study aims to build upon the investigators' previous evaluations of the program, and will provide insight into the ability of this program to serve youth at a much larger scale. Specifically, this study aims to answer the following research questions:
(1) What is the effect of implementing an evidence-based individualized tutoring program at larger scale?
(2) What is the relationship between the effect of the program and the scale at which the program is implemented?
Implementation sites are divided into two sets: sites in Chicago at which students are randomized to receive tutoring (hereby referred to as “scale-up” schools), and sites in Chicago and New York City where principals have primary discretion to choose which students receive tutoring (hereby referred to as “returning schools”). In addition to randomizing students into the Saga program, scale-up schools are also served by randomly selected tutors. The research team is having Saga over-recruit tutors as though they were implementing at larger than the intended scale in the scale-up schools. Investigators then randomly select one in three-and-a-half tutor applicants to continue through Saga’s standard hiring process, and positions at the scale-up schools are only filled by these randomly selected tutors. All study schools (both scale-up and returning) are implementing a third form of randomization: students in the Saga program are randomly assigned to tutors.
In order to study research question #1, investigators will take advantage of the power of random sampling of tutors and the random assignment into the Saga program in the scale-up schools to study scale up of this program without actually having to implement the program at a much larger scale. By comparing the outcomes of students randomly assigned to the Saga program to students randomly assigned to the control group in these schools, we will be able to rigorously estimate the effects of the program if it were being staffed by the tutors that would work for a program that is three-and-a-half times as large as the program currently operating in the scale-up schools.
To gain insight into research question #2 above, which seeks to determine the relationship between program scale and effects, tutors at all sites are ranked by Saga leadership based on relative expected quality. Because the research team is randomizing student pairs to tutors, we will be able to isolate the effect of value-add of each tutor in the Saga program. Combining this information with the Saga rankings of applicant quality, the researcher team will be able to examine tutor ranking and tutor effectiveness. Assuming that the program would hire tutors in the order of their ranking depending on the number of tutor slots they needed to fill, this analysis will shed light on the relationship between scale and effectiveness.
At the end of academic year (AY) 2016-17 and AY2017-18, researchers will answer the research questions by looking at various academic and behavioral outcomes accessed through administrative data from the Chicago Public Schools, New York City Department of Education, and Chicago Police Department. These outcomes include GPA, credits attempted, credits earned, and standardized test scores in Chicago and New York City, and crime involvement in Chicago. Researchers will also have access to data on tutor characteristics, which includes demographic information and information on math ability and teacher training, to assess the variation in the effectiveness of tutors and to examine the degree to which tutor characteristics correspond to different student outcomes. Lastly, researchers will receive programmatic attendance data from Saga, which includes information on how often students were present and which tutors they worked with each day. From this attendance data, we will be able to compute an “average tutor ranking”, which will be a weighted average of tutor ranking and how many days a student worked with a tutor of that ranking.
Researchers believe the proposed project will provide valuable insights into how to scale up what may turn out to be an extremely cost-effective way to improve educational outcomes for low-income youth. But perhaps more importantly, the hope is that this work will contribute to a broader understanding of how program scale affects program quality and how to systematically study this relationship. The investigators hope this project will create a "roadmap" for a growing body of research on the science of scale-up, guiding efforts of future researchers to understand the implications of scale-up for interventions with scarce inputs.
|