Data-Driven Instruction in Honduras: An Impact Evaluation of the EducAcción Promising Reading Intervention

Last registered on June 07, 2018

View Trial History

Pre-Trial

Trial Information

General Information

Title

Data-Driven Instruction in Honduras: An Impact Evaluation of the EducAcción Promising Reading Intervention

RCT ID

AEARCTR-0000780

Initial registration date

October 05, 2015

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published

October 05, 2015, 11:27 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated

June 07, 2018, 4:08 PM EDT

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Country

Honduras

Region

Central America

Primary Investigator

Name

Sarah Liuzzi

Affiliation

Mathematica Policy Research

Contact Primary Investigator

Other Primary Investigator(s)

PI Name

Chantal Toledo

PI Affiliation

Mathematica Policy Research

Contact Investigator

PI Name

Steven Glazerman

PI Affiliation

Mathematica Policy Research

Contact Investigator

PI Name

Nancy Murray

PI Affiliation

Mathematica Policy Research

Contact Investigator

PI Name

Sarah Humpage-Liuzzi

PI Affiliation

Mathematica Policy Research

Contact Investigator

Additional Trial Information

Status

On going

Start date

2014-10-01

End date

2018-09-30

Keywords

Education

Additional Keywords

formative assessment, standardized testing, early grade reading

JEL code(s)

I20

Secondary IDs

Abstract

As policymakers’ focus in developing countries shifts from school access to school quality, student assessment has come to the fore. Governments use assessment data not only to monitor educational progress, but to provide feedback aimed at improving teaching and learning. In this evaluation, we focus on the potential effects of formative assessment (FA), in which teachers use frequent assessments of their students to improve their teaching, and end-of-grade (EOG) summative assessments, which, like FAs, can be the basis for teachers and principals to adapt their teaching practices and curricula to better meet the needs of their students.

We undertake a randomized controlled trial (RCT) in 180 schools in Honduras, complemented by qualitative analysis, to estimate the impacts of providing print materials and pedagogical support to maximize teachers’ ability to take advantage of these assessments. To do this, we randomize schools into 3 experimental groups: EOG wrap-around support and FA with support (Group A), EOG wrap-around support only (Group B), and ongoing programming by Honduras’ Ministry of Education (MOE) (Group C, the control group). Group C has limited access to printed FA materials from the Ministry’s previous efforts to distribute them. All schools administer an EOG test, but only schools in Groups A and B receive the intervention’s training and wrap-around support. We collected baseline data in October and November 2014 at the end of the school year. We conducted random assignment in January 2015. The intervention began in June 2015 and we began baseline analysis in June 2015. We will collect follow-up data at the end of 2015 and 2016.

This design allows us to answer three main sets of research questions. First, we will compare the outcomes of teachers and students in schools assigned to group A to those in schools assigned to group B. This first comparison allows us to provide unbiased estimates of the impact of the FA intervention, holding constant the EOG wrap-around support intervention. Second, we will compare the outcomes of teachers and students in schools assigned to group B to those in schools assigned to group C. This comparison allows us to provide unbiased estimates of the impact of the EOG wrap-around support intervention compared to the ongoing programming provided by MOE and other organizations. Third, we will also be able to compare the outcomes of teachers and students in schools assigned to group A to those in schools assigned to group C. This comparison allows us to provide unbiased estimates of the impact of the combined use of EOG wrap-around support and FA with support, compared to the ongoing programming provided by MOE and other organizations. We will estimate each intervention’s impact on process outcomes (teachers’ access to assessment materials and results), intermediate outcomes (teachers’ use of assessment results to modify instruction), and final outcomes (learning, as measured by EOG tests and an independent assessment). We will estimate impacts at the end of the first and second years of the interventions (the 2015 and 2016 academic years). We will evaluate impacts on teaching methods at the end of both years, and will evaluate impacts on reading and math test scores at the end of the second year of the intervention.

This study will also include an implementation study which will help demonstrate how the interventions are implemented in reality, and if the implementation differed from the original plans. The implementation study will consist of gathering and analyzing qualitative and quantitative data on what interventions were implemented, and how. Qualitative data collection will consist of key informant interviews and focus groups. In addition, if we find that the intervention is effective, we will also conduct cost-effectiveness analysis.

A rigorous evaluation of the impact of these types of assessments in Honduras will contribute to a growing body of evidence on what works to improve early grade reading in primary schools, and more generally will test the effectiveness and costs of promising reading and education-access interventions. This evaluation is part of a five-year project to rigorously evaluate and cost United States Agency for International Development (USAID) investments in early literacy and access to education in conflict settings in Latin America and the Caribbean (LAC). Mathematica is currently leading multi-year studies in four countries: Guatemala, Honduras, Nicaragua and Peru. This intervention and the evaluation of the intervention are funded by USAID.

External Link(s)

Registration Citation

Citation

Glazerman, Steven et al. 2018. "Data-Driven Instruction in Honduras: An Impact Evaluation of the EducAcción Promising Reading Intervention." AEA RCT Registry. June 07. https://doi.org/10.1257/rct.780-5.0

Former Citation

Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Experimental Details

Interventions

Intervention(s)

We undertake a randomized controlled trial (RCT) in 180 primary schools in Honduras, complemented by qualitative analysis, to estimate the impacts of providing print materials and pedagogical support to maximize teachers’ ability to take advantage of these assessments in early primary grades. To do this, we randomize schools into 3 experimental groups: EOG wrap-around support and FA with support (Group A), EOG wrap-around support only (Group B), and ongoing programming by Honduras’ Ministry of Education (MOE) (Group C, the control group). Group C has limited access to printed FA materials from the Ministry’s previous efforts to distribute them. All schools administer an EOG test, but only schools in Groups A and B receive the intervention’s training and wrap-around support. We collected baseline data in October and November 2014 at the end of the school year. We conducted random assignment in January 2015. The intervention began in June 2015 and we began baseline analysis in June 2015. We follow the cohort of students who were in the second grade in the first year of the 2-year intervention.

This design allows us to answer three main sets of research questions. First, we will compare the outcomes of teachers and students in schools assigned to group A to those in schools assigned to group B. This first comparison allows us to provide unbiased estimates of the impact of the FA intervention, holding constant the EOG wrap-around support intervention. Second, we will compare the outcomes of teachers and students in schools assigned to group B to those in schools assigned to group C. This comparison allows us to provide unbiased estimates of the impact of the EOG wrap-around support intervention compared to the ongoing programming provided by MOE and other organizations. Third, we will also be able to compare the outcomes of teachers and students in schools assigned to group A to those in schools assigned to group C. This comparison allows us to provide unbiased estimates of the impact of the combined use of EOG wrap-around support and FA with support, compared to the ongoing programming provided by MOE and other organizations. If group B schools receive FA support in practice, we will also compare outcomes for groups A and B schools combined to group C outcomes.

Intervention Start Date

2015-06-01

Intervention End Date

2016-12-23

Primary Outcomes

Primary Outcomes (end points)

We will estimate each intervention’s impact on:
1) process outcomes (teachers’ access to assessment materials and results)
2) intermediate outcomes (teachers’ use of assessment results to modify instruction)
3) final outcomes (academic achievement, as measured by end of grade tests and an independent assessment).

We will study impacts on process and intermediate outcomes at the teacher and school levels, and will estimate final outcomes on learning at the student level. We will estimate impacts at the end of the first and second years of the interventions (the 2015 and 2016 academic years).

Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)

Secondary Outcomes (explanation)

Experimental Design

After identifying our sample of 180 schools, we randomly assigned each school to one of the three experimental groups described above (A, B or c) with equal probabilities. We implemented random assignment in two steps to ensure that the random assignment would result in three groups that were balanced geographically, in terms of baseline exposure to the interventions and in terms of baseline performance. First, we divided each of four departments' (states') schools into a high or low baseline use group, according to their score on an index measuring baseline use of formative and end of grade assessment data. This generated eight strata. Then, we sorted schools within each stratum by performance on the schools’ 2013 first grade reading end of grade test scores. We then created triplets of schools with similar test scores within each stratum. For example, the three schools with the best scores in the group of low exposure schools in Lempira would be one triplet, the next three schools would be another triplet, and so forth. In addition to improving balance across the three experimental groups on geographic areas, baseline exposure, and baseline performance, this approach is expected to improve the precision of our estimates.

Experimental Design Details

Differences in average outcomes will represent unbiased estimates of the interventions’ impacts on outcomes. Here, we present an example of how we will measure impacts on the final outcome of student learning. The regression model will control for chance differences between treatment groups in schools’, teachers’, or students’ baseline characteristics. These will include schools’ EOG reading scores at baseline, teachers’ participation in training on assessment at baseline, teachers’ use of assessment at baseline, and other variables. Including covariates in our regression model will allow us to increase the statistical power of the study. We will also include dichotomous variables for the strata used in random assignment. For each outcome we analyze, the model can be expressed as follows:
y_ist= a +beta_xis0 +gamma_zs0 +lambdaA*T_As +lambdaB*T_Cs +alpha_1...alpha_r-1 +eta_s +epsilon_ist

where y_ist is the outcome of interest (such as reading or math test score) for student i in school s at time t; t will take on values of 0, 1 or 2, we will evaluate the equation above after one (t = 1) and two (t = 2) years of implementation, controlling for baseline (t = 0) characteristics, and we will include dummy variables for the r strata used in random assignment, as represented by alpha. The vector xis0 represents the baseline characteristics of student i in school s, which may include age, gender, and baseline reading measures y, if available. The vector zs0 represents the baseline characteristics of the school s, such as school size, whether teachers teach multiple grades at once, or school-level baseline reading or math test scores measures y. The variables TAs and TCs are indicators equal to one for students in schools assigned the intervention A or C, respectively, and zero otherwise. The term eta_s is a school-specific error term (a group or cluster effect); epsilon_ist is a random error term for student i in school s observed at time t. The parameters lambdaA and lambdaC represent the impact of the FA and EOG interventions for the A versus B comparison and the negative of the impact of the EOG intervention for the B versus C comparison, respectively. Therefore, we will test the restriction that lambdaA = 0 and also test lambdaC = 0. We will also test whether there is a significant difference between intervention groups A and C by testing the restriction that lambdaA = lambdaC.
After collecting midline data, we will determine whether there is a meaningful difference in implementation between groups A and B. EducAcción staff will not provide group B schools training or materials in support of the use of FA, but teachers in group B schools may choose to increase their use of FA on their own as a strategy to respond to their EOG test results. If this is common, there will not be a meaningful difference between the intervention implemented in group A and B schools. If this occurs, we will test the difference between outcomes of schools in either group A or B to outcomes of schools in group C. In other words, we will substitute lambdaAB (representing being in A or B) for lambdaA and lambdaB, and test the hypothesis that lambdaAB = 0. We will make this determination before analyzing the final outcome data.

Randomization Method

Randomization was done in office by a computer using Stata's random number generation.

Randomization Unit

The unit of randomization is the school.

Was the treatment clustered?

Yes

Experiment Characteristics

Sample size: planned number of clusters

180 schools.

Sample size: planned number of observations

1800 students, 10 per school. Because some small schools will not have 10 students, we will sample some additional children in larger schools to achieve a sample size of 1800 students.

Sample size (or number of clusters) by treatment arms

60 schools in experimental group A (EOG wrap-around support and FA with support), 60 schools in experimental group B (EOG wrap-around support only) and 60 schools in experimental group C (ongoing programming by Honduras’ Ministry of Education, control group).

Minimum detectable effect size for main outcomes (accounting for sample design and clustering)

The minimum detectable effect size (MDES) calculations presented here focus on test score outcomes because learning is the primary outcome of interest for the evaluation. We use existing end-of-grade (EOG) test score data from 2012 to calculate EOG test score standard deviations and intracluster correlations for the areas where the intervention will take place. The main inputs for the MDES calculations are: (1) the percentage of variance explained by the use of covariates in the regression model (R2) is assumed to be 0.1 for schools and 0.5 for students; and (2) the intracluster correlation, which is a measure of the similarity of students between schools versus the total between- and within-school variation, is estimated to be 0.06 based on 2012 EOG reading scores. These calculations also assume 80 percent power, statistical significance at the 5 percent level, and a two-tailed test. We will compare each of two treatment groups to a third reference group. For this type of comparison, the Dunnett test is the most appropriate test to control Type I error across multiple contrasts. Therefore, we assume that a Dunnett test will be used in the three comparisons studied. Assuming a sample of 180 schools, we will have a student sample of 1,800 students, or 10 students per school.

Supporting Documents and Materials

IRB