|
Field
Power calculation: Minimum Detectable Effect Size for Main Outcomes
|
Before
We base our computation on List et al. (2011) and account for intracluster correlation in the calculation of the minimal detectable effect size. In our experimental setting, there on average 16 schools in each experimental condition. Each school on average contains 40.6 students. We computed an intracluster correlation of 0.12. In the analysis, we can control for characteristics of schools and students, which would decrease the intracluster correlation. With the conventional power of 0.8 and a significance level of 0.05, we are able to detect a treatment effect of 0.37 standard deviations or larger.
Details of calculation:
To calculate the minimum detectable effect size, we follow List et al. (2011). They show that in a clustered design, the minimum number of observations in each experimental group can be computed as follows:
n=2(t_(α/2)+t_β)²(σ/δ)²(1+(m-1)ρ)
This implies that the minimum detectable effect size is equal to:
δ=σ/√(n/(2(t_(α/2)+t_β)²(1+(m-1)ρ)))
Or the minimum detectable effect size expressed as a fraction of a standard deviation is equal to:
δ/σ=1/√(n/(2(t_(α/2)+t_β)²(1+(m-1)ρ)))
δ/σ=1/√(649.3/(2(1.96+0.84)²(1+(40.6-1)0.12)))=0.37
Reference
List, J., Sadoff, S. and Wagner, M. (2011), So you want to run an experiment, now what? Some simple rules of thumb for optimal experimental design, Experimental Economics 14, 439-457
|
After
We base our computation on List et al. (2011) and account for intracluster correlation in the calculation of the minimal detectable effect size. In our experimental setting, there on average 16 schools in each experimental condition. Each school on average contains 40.6 students. We computed an intracluster correlation of 0.12. In the analysis, we can control for characteristics of schools and students, which would decrease the intracluster correlation. With the conventional power of 0.8 and a significance level of 0.05, we are able to detect a treatment effect of 0.37 standard deviations or larger.
Details of calculation:
To calculate the minimum detectable effect size, we follow List et al. (2011). They show that in a clustered design, the minimum number of observations in each experimental group can be computed as follows:
n=2(t_(α/2)+t_β)²(σ/δ)²(1+(m-1)ρ)
This implies that the minimum detectable effect size is equal to:
δ=σ/√(n/(2(t_(α/2)+t_β)²(1+(m-1)ρ)))
Or the minimum detectable effect size expressed as a fraction of a standard deviation is equal to:
δ/σ=1/√(n/(2(t_(α/2)+t_β)²(1+(m-1)ρ)))
δ/σ=1/√(649.7/(2(1.96+0.84)²(1+(40.6-1)0.12)))=0.37
Reference
List, J., Sadoff, S. and Wagner, M. (2011), So you want to run an experiment, now what? Some simple rules of thumb for optimal experimental design, Experimental Economics 14, 439-457
|