Minimum detectable effect size for main outcomes (accounting for sample
design and clustering)
We aim for a sample of 1600 observations with half being selected through the randomization process. We are hoping for 40 students in each of 40 bins. If onboarding survey covariates and stratification controls for the bins explain 40 percent of the variation in students’ outcomes, we calculate that with 80 percent power (the probability of rejecting the null hypothesis of zero effect), the minimum detectable effect (MDE) is 0.11 standard deviations.
To contextualize, consider the outcome “choosing a major related to the internship.” If 20 percent of students in the control group select a related major, the standard deviation for this variable is 0.4. An effect size of 0.11 standard deviations corresponds to a 4.4 percentage point increase in the likelihood of choosing a related major.