Experimental Design Details
Phase 1: Pilot Study (October 2014 – January 2015)
In April 2014, the project collected language and math test scores of 3,652 students in grades 4 and 5, corresponding to nearly 45% of the students of these grades. The first phase of the experiment consisted of two information treatments plus a control group. After merging test score data with administrative data the final sample for this pilot phase contains 3,026 students. These students were randomly assigned into three groups: two treatment groups (each with 1,008 students) and one control group (1,010 students).
• Treatment Group 1: Individual student performance – We provided families with a one-page report card that shows their child’s performance, as well as their relative position to the average performance of students in the same grade and school. The information is essentially a percentile rank (e.g. 50th percentile), presented in a way that is highly salient to all families.
• Treatment Group 2: School average performance – We provided families with a one-page report card that shows the average score of their child’s school in comparison to the average score across all schools in Manizales.
• Control Group: No information was provided.
To conduct the pilot, we randomly selected within the three groups a subsample of 2,100 students (800 in each treatment group, plus 400 in the control group) to conduct the home visits.
Phase 2: Expanded Study (February 2015 – April 2016)
Phase 2, the expanded study, was conducted between February 2015 and April 2016. In June 2015 , the project collected language and math test scores of the 3,026 students who were randomly assigned to the treatment or control conditions in Phase 1. In addition, the project collected language and math test scores of 1,345 students who were in grade 4 in 32 schools targeted in Phase 1. These 1,345 new students were randomly assigned into three groups: two treatments (each with 448 students) and a control group (with 449 students). We modified the information treatments from Phase 1 to Phase 2:
• Treatment Group 1: Individual student performance in the school – We provided families with a one-page report card that shows their child’s performance, as well as their relative position to the average performance of students in the same grade and school. Also included with the report card was a list of suggestions for parents to engage with their children’s education.
• Treatment Group 2: Individual student performance in the city – We provided families with a one-page report card that shows their child’s performance, as well as their relative position to the average performance of students in the same grade in the entire city. The same list of suggestions provided to parents in Treatment Group 1 was provided to parents in Treatment Group 2.
• Control Group: No information was provided.
Data in the second phase include the 3,026 students in Phase 1 plus an additional 1,345 students in grade 4, for a total sample of 4,371. We conducted a second home visit in November 2015. During this home visit, we provided information to parents of the 2,016 students who had been assigned to either treatment group in Phase 1. The information provided was based on the modified treatment arms described above. We also provided information to the parents of the 896 students who had been assigned to either treatment group in Phase 2. These home visits followed the same format as the home visits in Phase 1.
Phase 3: Full Study (May 2016 – August 2017)
The third phase of the study is different to the pilot study in two main senses. First, we expand the sample to all students in the grades of interest. As we described, in 2014 and 2015 students from 31 schools in grades 4, 5, and 6 were included. Beginning in 2016, we will not only follow those students that are in grades 5 and 6, but we will also include all students in grades 4, 5 and 6 that are enrolled in one of the 31 schools. We will not be following students who were in grade 5 in Phase 1 and are in grade 7 in Phase 3. Second, we will add a family engagement component.
In May 2016 we administered language and math tests to all students in grades 4, 5 and 6 within the 31 schools. This sample includes all students who were assigned to one of the three information intervention groups in Phase 1 and 2. Students who did not participate in Phase 1 or Phase 2 were not assigned to an information intervention group.
In addition to continuing to collect follow-up information for the information intervention, we included a family engagement component designed to improve trust and communication between the family and the school. In mid 2016, teachers were randomly assigned to one of two groups:
• Treatment Group: Family Engagement Intervention – Teachers received the test score information collected in May 2016 for the students in their class. Teachers also received a list of suggestions to promote improve family-school engagement, with two components. First, teachers received suggestions for teachers on how to improve communication with families. Second, teachers received suggestions on how to encourage families to engage with their children’s education outside of the school. This is the same list of suggestions that is provided to parents in the treatment groups in Phase 2.
• Control Group: No Family Engagement Intervention – Teachers did not receive test score information or the list of suggestions to promote family-school engagement.
At the end of each academic term we administered language and math tests to all students in the sample.