Minimum detectable effect size for main outcomes (accounting for sample
design and clustering)
The computation is based on List et al. (2011) and accounts for intracluster correlation in the calculation of the minimal detectable effect size. In our experimental setting, there are on average 7 schools in each experimental condition. Each school has on average 44.75 students. Computation in Stata based on the post-test shows that the intracluster correlation in the final sample equals 0.1. In the analysis, this intracluster correlation can be reduced by controlling for baseline characteristics of schools and students. Using the conventional power of 0.8 and a significance level of 0.1, the calculation results in a minimal detectable effect size of 0.52 standard deviations in case we would not control for students’ characteristics.
Details of the calculation:
According to List et al. (2011), in a clustered design, the minimum number of observations in each experimental group can be computed as follows:
This implies that the minimum detectable effect size is equal to:
Or the minimum detectable effect size expressed as a fraction of a standard deviation is equal to:
List, J., Sadoff, S. and Wagner, M. (2011), So you want to run an experiment, now what? Some simple rules of thumb for optimal experimental design, Experimental Economics 14, 439-457