The Success for All Model of School Reform

Last registered on April 13, 2016

Pre-Trial

Trial Information

General Information

Title
The Success for All Model of School Reform
RCT ID
AEARCTR-0001136
Initial registration date
April 13, 2016

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
April 13, 2016, 10:32 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Region

Primary Investigator

Affiliation
MDRC

Other Primary Investigator(s)

PI Affiliation
MDRC

Additional Trial Information

Status
Completed
Start date
2010-10-01
End date
2015-09-30
Secondary IDs
Abstract
Improving the reading performance of at-risk elementary students is one of the greatest challenges confronting American education. Success for All (SFA), a program aimed at ensuring that every child learns to read well in the elementary grades, is one of the best-known school reform models. SFA emphasizes phonics instruction for beginning readers and comprehension for all students. It incorporates cross-grade ability grouping for reading, highly structured lessons that make extensive use of cooperative leaning, frequent assessments of students’ progress, tutoring, staff teams to solve problems that go beyond academics, professional development for teachers, and a program facilitator at each school. On the basis of the program's strong record of success, the Success for All Foundation (SFAF) received a scale-up grant through the U.S. Department of Education’s Investing in Innovation (i3) initiative to expand the intervention to hundreds of additional schools.

MDRC conducted an independent evaluation of the scale-up initiative, using a cluster random assignment design in which 37 schools in five school districts that participated in the scale-up effort were assigned at random to a program group (19 schools), which put in place the intervention, or to a control group (18 schools).which implemented "business as usual" reading instruction. This design supports causal impact findings for the average school assigned to SFA.

MDRC collected and analyzed data from a variety of sources — teacher logs, principal and teacher surveys, and rating sheets completed by the coaches — to examine the implementation experiences of the schools participating in the study. The evaluation indicates that while SFA was implemented with adequate fidelity at the large majority of schools that adopted it, resource constraints prevented some schools from putting in place some key components, including the program's computerized tutoring for students needing assistance beyond the classroom. The evaluation also compared instruction and other characteristics of program and control schools. Program schools were unique in placing students in cross-grade ability groups for reading, and teachers in program group schools used cooperative learning as an instructional method more frequently than their counterparts in control schools. The two groups of schools did not differ greatly along other dimensions that were measured.

The impact evaluation focuses on a primary analysis sample of students who enrolled in kindergarten in the study schools and remained in these schools over three years, through second grade. The confirmatory analysis found that in Year 3, SFA produced a positive and statistically significant impact on one measure of phonics knowledge; the impact on a second phonics measure was positive, but not statistically significant. In Year 3, students in SFA schools did not outperform their control-school counterparts on measures of fluency or comprehension.

An exploratory analysis indicates that for a group of special concern to policymakers and practitioners -- students who started school with low preliteracy skills--the program had notable third-year effects. For those second-graders who, at the start of kindergarten, scored in the bottom half of the sample in terms of their knowledge of the alphabet and their ability to sound out words, the program produced positive and statistically significant gains on measures of phonics skills, word recognition, and reading fluency; the impact on comprehension was also positive, although not statistically significant. The program did not significantly affect these outcomes for the subgroup of students who started kindergarten in the top half of the sample in terms of phonetic skills.

SFA did not produce statistically significant effects on the average reading performance of students in grades 3-5. There were no consistent patterns or significant findings of program impacts on special education identification rates and retention rates.

A cost analysis made use of the random assignment design within one case study district to assess the extent to which the district's SFA schools required additional resources, relative to those used for reading program in the control group schools. These additional costs were relatively modest: Direct expenditures for the school-based reading facilitator's time, after-school tutoring time, materials, and professional development were estimated to cost $119 more per student per year in SFA schools than in control group schools. Adding to this the additional time that SFA principals devoted to the program, additional time that coaches and teachers spent on training, the extra cost of space for storing SFA materials, and other factors, program group schools in this district spent about $227 worth of total resources per student per year more than control group schools to implement their respective reading programs.

Through the fourth year of the i3 grant, SFA was put in place in 447 new schools and reached an estimated 276,000 students. These numbers fell below SFAF's ambitious goals but represent a notable achievement in a period of staff layoffs and other cutbacks in many schools and districts.

Registration Citation

Citation
Quint, Janet and Pei Zhu. 2016. "The Success for All Model of School Reform." AEA RCT Registry. April 13. https://doi.org/10.1257/rct.1136-1.0
Former Citation
Quint, Janet and Pei Zhu. 2016. "The Success for All Model of School Reform." AEA RCT Registry. April 13. https://www.socialscienceregistry.org/trials/1136/history/7655
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
First implemented in 1987, Success for All combines a challenging reading program, whole-school reform elements, and an emphasis on continuous improvement in order to improve the reading skills of children in the elementary grades. Specific features include a K-6 reading program that emphasizes phonics for beginning readers and comprehension for all students; briskly paced lesson plans and cooperative learning; cross-grade ability grouping for reading; computerized and individual tutoring; staff committees to address issues and promote parent and community involvement; and ongoing professional development.

Intervention Start Date
2011-07-01
Intervention End Date
2014-06-30

Primary Outcomes

Primary Outcomes (end points)
Reading achievement, primary analysis: Woodcock-Johnson Letter-Word Identification (WJLWI), Woodcock-Johnson Word Attack (WJWA), Test of Word Reading Efficiency (TOWRE), Woodcock-Johnson Passage Comprehension (WJPC) scores

Reading achievement, secondary analysis: Gates-MacGinitie total score, Gates-MacGinitie Vocabulary test, Gates-MacGinitie Comprehension test, state reading tests

Special Education and Specific Learning Disability new identification rates, student retention rate

Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
MDRC employed a cluster random assignment research design in which 37 schools located in five districts were assigned at random: 19 to receive the program and the remaining 18 to constitute a “business-as-usual” control group.

The evaluation compared the reading outcomes of kindergartners in both sets of schools in the first year of the study, and of first-graders and second-graders in the next two years, using age-appropriate assessments that were administered individually to each child. The primary analysis sample consists of students who started kindergarten in the study schools in the 2011-2012 school year and remained in the schools through all three implementation years. In the program group, these are the students with the maximum possible exposure to the program. A secondary analysis was conducted of students who were tested in the spring of 2014 and who received varying amount of the SFA intervention, ranging from less than one year to three years. The impact results for this "spring " sample parallel those for the primary analysis sample and reflect the effects of SFA when taking student mobility into account. Finally, during the second and third years of the study, students in third through fifth grades were also assessed, this time in a group setting. Student scores on state standardized tests served as additional measures of reading skills for this "auxiliary sample" of somewhat older students. Students in the auxiliary sample were in grades 1 through 3 when the study began and therefore did not first learn to read "the SFA way."

Two types of analytic models are used for analyzing outcomes measured at different levels. For outcomes measured at the student level, such as reading test scores, the main impact estimation model is a two-level hierarchical linear model that accounts for the clustering of students within schools. For outcomes measured at the school level, such as schools' special education identification rate and retention rate, the estimation model is an ordinary least squares (OLS) regression model with the school as the unit of observation. Both types of models use data from all five study districts in a single analysis, treating districts as fixed effects in the model. Separate program impact estimates are obtained for each district and then are averaged across the five districts, with each district's estimate weighted in proportion to the number of SFA schools it has. Therefore, findings in this report represent the impact on student performance in the average SFA school within the five study districts. The results may not necessarily reflect what the program effect would be in the wider population of districts beyond the study sample.

.
Experimental Design Details
Randomization Method
Randomization was done in the MDRC office by a computer.
Randomization Unit
Schools were randomly assigned within school districts.
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
37 schools
Sample size: planned number of observations
1,635 students in primary analysis sample
Sample size (or number of clusters) by treatment arms
19 treatment schools, 18 control schools
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
For the full primary analysis sample: WJLWI: 0.17 SD WJWA: 0.18 SD TOWRE: 0.18 SD WJPC: 0.17 SD
IRB

Institutional Review Boards (IRBs)

IRB Name
MDRC IRB
IRB Approval Date
2010-12-16
IRB Approval Number
NONE

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Yes
Data Collection Completion Date
Final Sample Size: Number of Clusters (Unit of Randomization)
37 schools (no attrition)
Was attrition correlated with treatment status?
Final Sample Size: Total Number of Observations
1,631
Final Sample Size (or Number of Clusters) by Treatment Arms
19 SFA schools, 18 control schools
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
No
Reports, Papers & Other Materials

Relevant Paper(s)

Abstract
First implemented in 1987, the Success for All (SFA) school reform model combines three basic elements:
--Reading instruction that is characterized by an emphasis on phonics for beginning readers and comprehension for students at all levels, a highly structured curriculum, an emphasis on cooperative learning, across-grade ability grouping and periodic regrouping, frequent assessments, and tutoring for students who need extra help
--Whole-school improvement components that address noninstructional issues
--Strategies to secure teacher buy-in, provide school personnel with initial and ongoing training, and foster shared school leadership

Success for All was selected to receive a five-year scale-up grant under the U.S. Department of Education’s first Investing in Innovation (i3) competition. This report, the first of three, examines the program’s implementation and impacts in 2011-2012, the first year of operation, at 37 kindergarten through grades 5 and 6 (K-5 and K-6) schools in five school districts that agreed to be part of the scale-up evaluation: 19 “program group” schools were randomly selected to operate SFA, and 18 “control group” schools did not receive the intervention. Program and control group schools were very similar at the start of the study. The analysis compares the experiences of school staff as well as the reading performance of a cohort of kindergarten students who remained in SFA schools throughout the year (and therefore received the maximum “dosage” of the program) with those of their counterparts in the control group schools.

Key Findings

--While teachers in the SFA schools initially expressed concerns about implementing this new, complex, and demanding initiative, by the end of the first year, many teachers were beginning to feel more comfortable with the program.
--Almost all the program group schools had reached a satisfactory level of early implementation as determined by the Success for All Foundation, the nonprofit organization that provides materials, training, and support to schools operating the reform. Yet there was also ample room for schools to implement additional program elements and to refine the elements that they had put in place.
--Reading instruction in the two sets of schools was found to differ in key ways.
--Kindergartners in the SFA schools scored significantly higher than their control group counterparts on one of two standardized measures of early reading. The impact on this measure seems to be robust across a range of demographic and socioeconomic subgroups, as well as across students with different levels of literacy skills at baseline.

Subsequent reports will examine the reading skills of these students as they progress through first and second grades and will also measure the reading skills of students in the upper elementary grades.
Citation
Quint, Janet C., Rekha Balu, Micah DeLaurentis, Shelley Rappaport, Thomas J. Smith, and Pei Zhu. 2013. The Success for All Model of School Reform: Early Findings from the Investing in Innovation (i3) Scale-Up. New York: MDRC.
Abstract
First implemented in 1987, Success for All (SFA) is a whole-school reform initiative whose goal is to help all elementary school students become competent readers. Its key elements include:
--Reading instruction marked by an emphasis on phonics and on comprehension, a highly structured curriculum, use of cooperative learning strategies, across-grade ability grouping,
frequent assessments, and tutoring for students who need extra help
--Components that address students’ noninstructional issues
--Strategies to secure teacher buy-in, provide teachers and leaders with initial and ongoing training, and foster shared leadership

SFA was selected to receive a five-year scale-up grant under the U.S. Department of Education’s Investing in Innovation (i3) competition. This, the second of three major reports from MDRC’s independent evaluation of the scale-up effort, discusses the program’s implementation and impacts in 2012-2013, the second year of operations at the study sites. The impact evaluation uses a cluster random assignment design involving 37 schools serving students in kindergarten through grades 5 or 6 (K-5 or K-6) and located in five school districts; 19 schools were randomly selected to receive the SFA program, while the remaining 18 “control group” schools did not receive the intervention. The report considers the experiences of school staff members and compares the reading performance in first grade of a cohort of kindergarten students who remained either in the SFA schools or in the control group schools for two years (and therefore received the maximum program “dosage”).

Key Findings
--During the second year, schools strengthened their implementation of SFA. New program practices were implemented, and the proportion of classrooms within a school where SFA-prescribed practices were in evidence increased. Teachers also reported feeling more comfortable with the program.
--By the end of the second year, 16 of the 19 program group schools were judged to meet the Success for All Foundation’s standards for adequate implementation fidelity.
--SFA reading classes continued to be distinguished from reading classes in the control group schools by greater use of cooperative learning, more extensive ability grouping of students, and close adherence to the curriculum. Tutoring, although a key program element, was not implemented more widely in SFA schools than in control group schools.
--First-graders who had been enrolled in SFA schools since kindergarten significantly outperformed their counterparts who had been continuously enrolled in control group schools on two measures of phonetic and decoding skills, although not on measures of fluency and comprehension, which are higher-order reading skills. Impact findings for subgroups of students defined by various demographic characteristics are, for the most part, consistent with the main findings.
Citation
Quint, Janet C., Rekha Balu, Micah DeLaurentis, Shelley Rappaport, Thomas J. Smith, and Pei Zhu. 2014. The Success for All Model of School Reform: Interim Findings from the Investing in Innovation (i3) Scale-Up. New York: MDRC.
Abstract
Success for All (SFA) is one of the best-known school reform initiatives. Combining a challenging reading program, whole-school reform elements, and an emphasis on continuous improvement, it seeks to ensure that every child learns to read well in the elementary grades. In 2010, the Success for All Foundation (SFAF) received a scale-up grant under the U.S. Department of Education’s Investing in Innovation (i3) program. This third and final report from the independent evaluation of the i3 scale-up examines the program’s implementation and impacts over three years, its incremental cost, and the scale-up process itself. Thirty-seven evaluation schools in five school districts were randomly assigned either to a program group of 19 schools that received SFA or to a control group of 18 schools that used alternative reading programs. This design supports causal impact findings for the average school assigned to SFA. Overall, the evaluation led to several key findings:

--Although SFA was implemented with adequate fidelity at the great majority of schools that adopted it, resource constraints prevented some schools from putting in place some of its key features, including a full-time facilitator and SFA’s computerized tutoring program.

--Program group and control group schools were different in some respects (for example, SFA schools were unique in placing students in cross-grade ability groups for reading, and SFA teachers made greater use of cooperative learning) but similar in others.

--SFA is an effective vehicle for teaching phonics. In the average SFA school, the program registered a notable, statistically significant impact on a measure of phonics skills for second-graders who had been in SFA for all three years, compared with their control group counterparts. Students in the average SFA school did not outperform their counterparts in the average control group school on tests of reading fluency or comprehension.

--For a subgroup of special concern to policymakers and practitioners — students entering school with low preliteracy skills — SFA appears to be especially effective. Second-graders in the average SFA school who had started kindergarten in the bottom half of the sample in terms of their knowledge of the alphabet and their ability to sound out words registered significantly higher scores on measures of phonics skills, word recognition, and reading fluency than similar students in control group schools. The impact on comprehension for this group was also positive but not statistically significant. The program did not significantly affect outcomes for the subgroup of students who started kindergarten in the top half of the sample in terms of phonetic skills.

--In a case study district, the direct expenditures for additional reading facilitator time, after-school tutoring, materials, and professional development were estimated to cost $119 more per student per year in SFA schools than in control group schools. Including the extra time that SFA principals devoted to the program and that coaches and teachers spent in training, the extra cost of space for storing SFA materials, and other factors, program group schools spent about $227 worth of resources per student per year more than control group schools.

--Through the fourth year of the i3 grant, SFA was put in place in 447 new schools and reached an estimated 276,000 students. These numbers fell below SFAF’s ambitious goals but represent a notable achievement in a period of staff layoffs and other cutbacks in many schools and districts.
Citation
Quint, Janet C., Pei Zhu, Rekha Balu, Shelley Rappaport, Micah DeLaurentis. 2015. Scaling Up the Success for All Model of School Reform: Final Report from the Investing in Innovation (i3) Evaluation. New York: MDRC.

Reports & Other Materials