Minimum detectable effect size for main outcomes (accounting for sample
design and clustering)
For the survey outreach experiment, the planned sample includes approximately 7,000 state legislators. For compensation arms, with approximately 1,400 legislators per arm, a two-sided 5% significance level, 80% power, and a baseline response rate of 15%, the minimum detectable difference for a pairwise comparison between two compensation arms is approximately 3.8 percentage points. For comparisons between one of the four larger follow-up arms, with approximately 1,680 legislators per arm, the minimum detectable difference is approximately 3.5 percentage points. For comparisons involving the smaller phone-call arm, with approximately 280 legislators, the minimum detectable difference is approximately 6 percentage points, depending on the comparison group.
Power for the vignette experiment depends on the realized survey response rate. If the response rate is between 15% and 25%, the expected number of respondents will be approximately 1,050 to 1,750. With equal assignment across the four vignette arms, this implies approximately 260 to 440 respondents per vignette arm. For the main effects in the 2 × 2 vignette design, this provides 80% power at the 5% significance level to detect effects of approximately approximately 8.6 percentage points for binary outcomes with a mean near 0.50. Detectable effects for pairwise comparisons between individual vignette arms and for interaction effects will be larger. These calculations are approximate and do not account for precision gains from covariate adjustment or randomization-strata fixed effects.