Experimental Design Details
Differences in average outcomes will represent unbiased estimates of the interventions’ impacts on outcomes. Here, we present an example of how we will measure impacts on the final outcome of student learning. The regression model will control for chance differences between treatment groups in schools’, teachers’, or students’ baseline characteristics. These will include schools’ EOG reading scores at baseline, teachers’ participation in training on assessment at baseline, teachers’ use of assessment at baseline, and other variables. Including covariates in our regression model will allow us to increase the statistical power of the study. We will also include dichotomous variables for the strata used in random assignment. For each outcome we analyze, the model can be expressed as follows:
y_ist= a +beta_xis0 +gamma_zs0 +lambdaA*T_As +lambdaB*T_Cs +alpha_1...alpha_r-1 +eta_s +epsilon_ist
where y_ist is the outcome of interest (such as reading or math test score) for student i in school s at time t; t will take on values of 0, 1 or 2, we will evaluate the equation above after one (t = 1) and two (t = 2) years of implementation, controlling for baseline (t = 0) characteristics, and we will include dummy variables for the r strata used in random assignment, as represented by alpha. The vector xis0 represents the baseline characteristics of student i in school s, which may include age, gender, and baseline reading measures y, if available. The vector zs0 represents the baseline characteristics of the school s, such as school size, whether teachers teach multiple grades at once, or school-level baseline reading or math test scores measures y. The variables TAs and TCs are indicators equal to one for students in schools assigned the intervention A or C, respectively, and zero otherwise. The term eta_s is a school-specific error term (a group or cluster effect); epsilon_ist is a random error term for student i in school s observed at time t. The parameters lambdaA and lambdaC represent the impact of the FA and EOG interventions for the A versus B comparison and the negative of the impact of the EOG intervention for the B versus C comparison, respectively. Therefore, we will test the restriction that lambdaA = 0 and also test lambdaC = 0. We will also test whether there is a significant difference between intervention groups A and C by testing the restriction that lambdaA = lambdaC.
After collecting midline data, we will determine whether there is a meaningful difference in implementation between groups A and B. EducAcción staff will not provide group B schools training or materials in support of the use of FA, but teachers in group B schools may choose to increase their use of FA on their own as a strategy to respond to their EOG test results. If this is common, there will not be a meaningful difference between the intervention implemented in group A and B schools. If this occurs, we will test the difference between outcomes of schools in either group A or B to outcomes of schools in group C. In other words, we will substitute lambdaAB (representing being in A or B) for lambdaA and lambdaB, and test the hypothesis that lambdaAB = 0. We will make this determination before analyzing the final outcome data.