Minimum detectable effect size for main outcomes (accounting for sample
design and clustering)
We conduct power analysis using optimal design software aiming for a power of 80% and a false discovery rate (alpha) of 5%.
For H1, we compare the pooled T1 & T2 (N = 8000) to the control group (N=4000). This sample gives us a MDES of 0.06 standard deviations after correcting for the imbalanced group size.
For H2, we compare the pooled T3 (N = 13 000) to the control group (N=4000). This sample gives us a MDES of 0.05 standard deviations after correcting for the imbalanced group size.
For H3, we compare the T1 (N = 4 000) to the control T2 (N=4000). This sample gives us a MDES of 0.064 standard deviations.
For H4, we compare the T3a (N = 7 000) to the control T3b (N=6000). This sample gives us a MDES of 0.05 standard deviations.
For H5, we compare the pooled T1 & T2 (N = 8000) to the pooled T3 (N=13 000). This sample gives us a MDES of 0.04 standard deviations after correcting for the imbalanced group size.
For the survey outcomes, we, conservatively, expect a response rate of 10%. Adjusting the sample size accordingly, assuming equal participation accross treatments leads to the following MDES using the camparisons outlined above.
For H1, we compare the pooled T1 & T2 (N = 800) to the control group (N=400). This sample gives us a MDES of 0.19 standard deviations after correcting for the imbalanced group size.
For H2, we compare the pooled T3 (N = 1 300) to the control group (N=400). This sample gives us a MDES of 0.16 standard deviations after correcting for the imbalanced group size.
For H3, we compare the T1 (N = 400) to the control T2 (N=400). This sample gives us a MDES of 0.20 standard deviations.
For H4, we compare the T3a (N = 700) to the control T3b (N=600). This sample gives us a MDES of 0.16 standard deviations.
For H5, we compare the pooled T1 & T2 (N = 800) to the pooled T3 (N=1300). This sample gives us a MDES of 0.13 standard deviations after correcting for the imbalanced group size.
We expect to improve power with the addition of control variables