Abstract
Improving the reading performance of at-risk elementary students is one of the greatest challenges confronting American education. Success for All (SFA), a program aimed at ensuring that every child learns to read well in the elementary grades, is one of the best-known school reform models. SFA emphasizes phonics instruction for beginning readers and comprehension for all students. It incorporates cross-grade ability grouping for reading, highly structured lessons that make extensive use of cooperative leaning, frequent assessments of students’ progress, tutoring, staff teams to solve problems that go beyond academics, professional development for teachers, and a program facilitator at each school. On the basis of the program's strong record of success, the Success for All Foundation (SFAF) received a scale-up grant through the U.S. Department of Education’s Investing in Innovation (i3) initiative to expand the intervention to hundreds of additional schools.
MDRC conducted an independent evaluation of the scale-up initiative, using a cluster random assignment design in which 37 schools in five school districts that participated in the scale-up effort were assigned at random to a program group (19 schools), which put in place the intervention, or to a control group (18 schools).which implemented "business as usual" reading instruction. This design supports causal impact findings for the average school assigned to SFA.
MDRC collected and analyzed data from a variety of sources — teacher logs, principal and teacher surveys, and rating sheets completed by the coaches — to examine the implementation experiences of the schools participating in the study. The evaluation indicates that while SFA was implemented with adequate fidelity at the large majority of schools that adopted it, resource constraints prevented some schools from putting in place some key components, including the program's computerized tutoring for students needing assistance beyond the classroom. The evaluation also compared instruction and other characteristics of program and control schools. Program schools were unique in placing students in cross-grade ability groups for reading, and teachers in program group schools used cooperative learning as an instructional method more frequently than their counterparts in control schools. The two groups of schools did not differ greatly along other dimensions that were measured.
The impact evaluation focuses on a primary analysis sample of students who enrolled in kindergarten in the study schools and remained in these schools over three years, through second grade. The confirmatory analysis found that in Year 3, SFA produced a positive and statistically significant impact on one measure of phonics knowledge; the impact on a second phonics measure was positive, but not statistically significant. In Year 3, students in SFA schools did not outperform their control-school counterparts on measures of fluency or comprehension.
An exploratory analysis indicates that for a group of special concern to policymakers and practitioners -- students who started school with low preliteracy skills--the program had notable third-year effects. For those second-graders who, at the start of kindergarten, scored in the bottom half of the sample in terms of their knowledge of the alphabet and their ability to sound out words, the program produced positive and statistically significant gains on measures of phonics skills, word recognition, and reading fluency; the impact on comprehension was also positive, although not statistically significant. The program did not significantly affect these outcomes for the subgroup of students who started kindergarten in the top half of the sample in terms of phonetic skills.
SFA did not produce statistically significant effects on the average reading performance of students in grades 3-5. There were no consistent patterns or significant findings of program impacts on special education identification rates and retention rates.
A cost analysis made use of the random assignment design within one case study district to assess the extent to which the district's SFA schools required additional resources, relative to those used for reading program in the control group schools. These additional costs were relatively modest: Direct expenditures for the school-based reading facilitator's time, after-school tutoring time, materials, and professional development were estimated to cost $119 more per student per year in SFA schools than in control group schools. Adding to this the additional time that SFA principals devoted to the program, additional time that coaches and teachers spent on training, the extra cost of space for storing SFA materials, and other factors, program group schools in this district spent about $227 worth of total resources per student per year more than control group schools to implement their respective reading programs.
Through the fourth year of the i3 grant, SFA was put in place in 447 new schools and reached an estimated 276,000 students. These numbers fell below SFAF's ambitious goals but represent a notable achievement in a period of staff layoffs and other cutbacks in many schools and districts.