Remedial Science Education

The proposed research project will use random assignment to evaluate the impact of a remedial education program for low-performing third-grade science students in 48 schools in metropolitan Lima, Peru.

2014-06-05

2014-11-28

Key evaluation outcome measures are: i) students’ understanding of science and the environment and ii) performance on achievement tests.

Randomization will take place at the student level, with a separate lottery for each participating school. All third grade students from a given school will take the baseline exam that will identify students that place in the lowest half in each school. The evaluation will target 48 schools in metropolitan Lima that participated in either treatment or control groups in the 2012 pilot project.

The evaluation will target low-performing third-grade students from 48 schools in metropolitan Lima that participated in either treatment or control groups in the 2012 pilot project. Within each school, half of the low-performing students will be assigned to the remedial education treatment and half to control conditions. The evaluation design will therefore stratify at the school level.

According to total enrollment levels collected from school principals at baseline in 2012, we anticipate, on average, 48 third-grade students per school distributed in two third-grade sections per school. Logistical reasons explain why we aim to target the same schools as the 2012 pilot. In the treatment schools, UCH already has contacts with the administration, which should facilitate the implementation of the program. In addition, data collection by IPA in the control schools went according to plan and there was little attrition and non-response.

IPA Peru will administer a baseline evaluation test to students in April 2014. This test will be a similar though simplified version of the science test that was used in the 2012 Science Education pilot. This test was developed to assess third-grade science proficiency according to the Peruvian National Curriculum, which includes as subdomains: “Physical World” (“Mundo Físico y conservación del ambiente”), “Human body and health” (“Cuerpo humano y conservación de la salud”) and “Animals and environment conservation” (“Animales y ambiente”). It is a timed self-administered written test that students will be given to complete on their own, without a surveyor. A surveyor will be present at the school to supervise the appropriate and timely administration of the test. Each section (corresponding to one of the three areas of the curriculum) is separately timed.

During the implementation of the program, IPA Peru will develop an end-line test of the same abilities measured for the baseline. The test will be given at the end of the school year to all the third grade students in participating schools to see if those who were placed in the treatment group were able to close the gap with the other students from their class. Both baseline and end-line will be norm-referenced, standardizing all student test scores in standard deviations above or below the control group mean.

In addition, teachers will be surveyed to collect demographic data and to gather their views on different aspects of teaching Science and Environment. Collecting this information at baseline and end-line will permit observing any changes in teacher’s perceptions of teaching science material. School directors of the sample schools will also be surveyed about their experience and characteristics of their schools, including the number of teachers, school enrollment, infrastructure, equipment and learning materials and school climate. Lastly, students will be sent home with questionnaires for parents with questions about the support and assistance that the child receives for homework, the household's socioeconomic status and other characteristics such as household size and education of parents. Both the data collected at baseline from the school directors and parents will serve to ensure balance between treatment and control groups. Additionally, the data will also be used to perform heterogeneity analysis to identify variation in effects between subgroups, such as students in homes with a working television, or students in schools with operational sanitation facilities

Randomization done in office by a computer, coin flip etc.

Randomization units are individual third-grade students who are low-performing according to baseline test. Randomization will be stratified, with one lottery per school (48 schools= 48 lotteries).

No

Randomization is at the student level, stratified at the school level. There are 48 schools participating in this trial

48 schools, 1100 students, 100 teachers.

48 schools, 1100 students, 100 teachers.

As the Targeted Science Education program will be working with the same schools in the metropolitan Lima area as did the 2012 science education pilot, power calculations rely on the 2012 sample. Specifically, we expect that among the 48 participating schools in metropolitan Lima, there will be approximately 1100 students that place in the lowest half of the baseline achievement distribution. Among these 1100 students, half will be randomly assigned to treatment and the other half will be assigned to control conditions. The attrition rate during the 2012 Science Education pilot was 13 percent, and a more conservative 15 percent is used for these power size calculations. A correlation of 0.576 between baseline and end line was calculated, based on the 2012 sample of students. Assuming a five percent level of significance and 80 percent statistical power, the expected minimum detectable treatment effect is (MDE) 0.16 standard deviations. As the randomization will be stratified by school, this reduces the MDE holding all other parameters constant, making this MDE likely a conservative estimate. We find it is reasonable to conduct an empirical study in which the MDE is 0.16 given the results of previous remedial and tracking programs, ranging from .14 to .28 standard deviations.