Minimum detectable effect size for main outcomes (accounting for sample
design and clustering)
To determine the appropriate sample size for our study, we conducted an a priori power analysis using G*Power 3.1 (Faul et al., 2009). Given our study design, we aimed to detect a medium effect size with a power of 0.80 and an alpha level of 0.05.
We planned to compare mean differences in four outcome variables (support for funding proposed municipal project, support for residential property tax as a funding source, voting preference, and ranking of preferred funding sources) across four treatment conditions (control, image, gain-framed, loss-framed), while controlling for covariates (both continuous and categorical, e.g., socio-demographic factors, primary residence type and ownership, political leaning, partisanship, knowledge of municipal finance, and trust in local government). Given this, we conducted a power analysis for a Multivariate Analysis of Variance (MANOVA) and a Linear Multiple Regression.
Power Analysis for MANOVA
We used the “MANOVA: Repeated measures, between factors” option in G*Power to estimate the required sample size. The parameters were set as follows:
Effect size (f) = 0.25 (medium, based on Cohen, 1988, 1992) / 0.10 (small effect)
Number of groups = 4 (treatment conditions)
Number of measures = 4 (outcome variables)
Corr among repeated measures = 0.8
Alpha level (α) = 0.05
Power (1-β) = 0.80
The analysis estimated a required sample size of 156 participants (if medium effect) (or 932 if small effect) to achieve sufficient power.
Power Analysis for Linear Multiple Regression
Given that we also planned to conduct an Ordinary Least Squares (OLS) regression with four outcome variables as the dependent variables, treatment dummies as independent variables, and a set of covariates, we conducted an additional “Linear multiple regression: Fixed model, R² deviation from zero” analysis in G*Power:
Effect size (f²) = 0.15 (medium) / 0.02 (small effect)
Number of predictors = 23 (3 treatment dummies + 20 covariates)
Alpha level (α) = 0.05
Power (1-β) = 0.80
This analysis suggested a lower sample size of 166 participants (if medium effect) (or 1,124 if small effect).
Final Sample Size Decision
This power analysis ensures adequate sensitivity to detect meaningful treatment effects while controlling for key covariates. We used conservative parameter estimates in the power calculation and adopted the more conservative estimate for final sample size. The targeted sample size of 1,500 respondents ensures sufficient statistical power for this experimental study.
References
Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009). Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41, 1149-1160.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, New Jersey: Lawrence Erlbaum Associates.
Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155–159.