Experimental Design
The Servicio País en Educación (SPE) program emerged from a partnership between the Chilean Ministry of Education (Mineduc) and the Fundación para la Superación de la Pobreza (Poverty Alleviation Foundation, FSP) and was first implemented during the period between September and December of 2010. The main objective of the program was to improve both attitudes toward reading (IR) and reading comprehension (RC) in 4th graders from vulnerable schools. Additionally, the program aimed at intervening in two additional dimensions: to generate new learning environments in which to implement the program, being this aspect influenced by the earthquake that affected the central region of Chile in February, 2010. Secondly, SPE intended to include college students as direct intervention agents. However, this evaluation only answers the questions related to the impact of the program on IR and RC.
The SPE program targeted vulnerable schools of the central region of Chile. Additionally, a number of logistic restrictions were put by FSP. Researchers excluded all schools from counties in which the FSP either was not able to work because they had no human resources in them or had already committed with some schools in it, which made randomization impossible there. This reduced the number of counties from which schools were included in the evaluation to 10: Santiago, Estación Central, Lo Espejo, Maipú, La Florida and San Bernardo from the Great Santiago region (Metropolitan Region, RM), and Concepción, Coronel, Hualpén and Talcahuano from the Biobio Region (VIII). In some of those counties, an additional restriction was set in terms of the administrative dependence of the schools, restricting us to include either only public schools (P) in some counties or only private subsidized schools (PS) in other ones. In addition, to fit the operational model designed by the FSP, each of the included schools had to have at least 90 students in fourth grade.
Using this sample, schools were randomly assigned to treatment and control groups, stratifying by county, socioeconomic group and SIMCE test scores. As the eligible school set was larger than the number of required schools, only some of the schools assigned to each group were included in the evaluation, decision that was random too. The remaining schools were kept as replacement lists for the eventual rejection of schools to take part of the evaluation. With the results of this assignment, schools were contacted and invited to take part of the evaluation in their corresponding group. Five of them rejected the program but, except in two cases, all of them accepted to be evaluated anyway. Additionally, two schools in the control group rejected to be evaluated. All of these schools were randomly replaced by another schools coming from the replacement lists. The treatment and control groups were finally composed of 45 and 40 schools respectively, grouped in 25 and 24 units.
Baseline data on the students were collected before the start of the program in August 2010, and follow-up data were collected after the program finished in December 2010. Researchers included two instruments. (1) The Prueba de Comprensión Lectora y Producción de Textos (Reading Comprehension and Texts Production Test, CLPT) measures formal reading skills, i.e., Reading Comprehension (RC), Texts Production (TP), and use of Language (UL). (2) The Gusto por la Lectura (Taste for Reading, GPL) measures attitudes towards reading along the four dimensions: Interest for Reading (IR), Self- perception as a Reader (SPR), Enjoyableness for Reading (ER) and Perception of Reading at School (PRS). These four indexes move discretely between 0 and 3, where 0 is the most negative of the offered alternatives and 3 is the most positive of the offered alternatives. Researchers also collected information on the program operation to understand the reasons behind the heterogeneous impacts of the program in different subpopulations.
First, researchers monitored the implementation of the program with random visits to observe the actual tutoring sessions. Second, they gathered administrative information about student, tutor, and professional assistance to tutoring sessions and about the numbers of sessions received by each student in the program. Finally, researchers collected data from the Ministry of Education on both schools’ and students’ and characteristics. Regarding schools, researchers collected data on Language and Math SIMCE test scores, average mothers’ years of schooling, average household income, socioeconomic level, a school vulnerability index (IVE), and administrative dependence. Regarding students, researchers collected information on gender, grades for 2009 and school attendance for 2009 and 2010, coming them from Mineduc records too.
Researchers estimated the effect of the program by comparing average outcomes of the treatment and the control group. Additionally, they perform some statistical exercise in order to understand the relationship between the two tests results. To assess the Intention-to-Treat (ITT) and the Treatment-on-the-Treated (TT) effects on the CLPT scores, researchers use OLS models. For TT, researchers use number of sessions of the program received by a student, which stands as a measure of the intensity of the program, and which researchers instrument using the intention-to-treat dummy as an instrumental variable (IV). To estimate the ITT and TT effects of the program on GPL test scores, researchers used ordered logit models. To estimate the TT effect, researchers used the number of sessions of the program received by the student to control for endogeneity. Researchers also present cost effectiveness analyses.