Minimum detectable effect size for main outcomes (accounting for sample
design and clustering)
Based on earlier CAL research, we aim to be able to detect an ITT minimum effect of increasing mathematical achievement by 0.10s (standard deviations) given a 50% program take-up rate (or 0.20s among takers). The power calculations that we present next are for the comparison between TA1 and the Control group at the end of Year 2 of the study.
Data on PRDE student test scores from previous years shows an intra-cluster correlation of 0.12 at the school level (ρ=0.12). Our power calculations consider 80 students and 2.4 Grade 4-8 math teachers per school, on average, with 112 schools in each Treatment Arm and 224 in the Control group. We assume that the outcome variable is standardized within the test-taking population and that after controlling for baseline scores, the residual standard deviation equals 0.9 (sd=0.9). Given this cluster-randomized design, power calculations for ITT effects (power=0.8, α=0.05) indicate that the MDE of comparing TA1 and the control group at the end of year 2 is 0.106s. Our power analysis is conservative as we will use other baseline variables to reduce the outcome's residual variance.
We also perform power calculations for the short-term effects of the program. First, we calculate MDE for the comparison between schools in TA1 and the comparison group (the control group, TA3, and TA4) in Year 1 of the program. We use the same parameters for the intra-cluster correlation (ρ=0.12), and the standard deviation of the outcome (sd=0.9). The MDE of this comparison is 0.096s. Second, the MDE for comparing TA3 and the control group two years after the start of the program is 0.106s. Again, this power analysis is conservative as we will use other baseline variables to reduce the outcome's residual variance.
Given that our sample includes all primary and middle schools in Puerto Rico, we have statistical power to identify reasonably small MDEs. Furthermore, pooling Year 1 data for students in Cohort 1 and Year 2 data for students in Cohort 2 will allow us to gain significantly greater precision to detect short-term one-year ITT estimates of the interventions.