Minimum detectable effect size for main outcomes (accounting for sample
design and clustering)
The minimum detectable effect size depends on the fraction of inattentive respondents that will be discarded out of the 1680 observations per treatment. Assuming less than 5% of the respondents are discarded (based on pilot data), we are left with about 1600 per treatment. Assuming a 80% power and a 5% threshold for the normal distribution:
- A 5 percentage point cross-treatment difference in a dichotomous variable shall be detectable (ex: 20 vs 25% of people signing the petition between two treatments: [(0.2*0.8+0.25*0.75)/sqrt(0.05^2)]*7.9 = 1,100 per treatment).
an effect size of 5p.p. between 20% and 25% across two treatments is detectable, a 4% is not (would need 1690 subjects).
- For a continuous outcome, for instance in the knowledge score, a difference as small as 0.12 (score from 0 to 5), with a standard deviation of 1.2 (based on pilot data) should be detectable (7.9*2/((0.12/1.2)^2)=1,580). With a standard deviation of up to 2, the difference shall be at least 0.2 to be detectable (1,580 respondents).