Minimum detectable effect size for main outcomes (accounting for sample
design and clustering)
We calculate the minimum detectable treatment effect sizes for the two primary outcome variables based on the prespecified empirical strategy. We calculate minimum detectable effects (MDEs) on a range of baseline response rates between 0.25% and 5%. When necessary, we pick 0.34% and 2% for low and high response-rate scenarios, based on baseline results. For the log income outcome variable, we assume the baseline mean of 11.07 with the standard deviation of 1.01, based on summary statistics from the pilot data. We calculate these MDEs for a given pair of treatment arms, which could be as small as approximately 6,600 recipients each, or as large as approximately 11,000 and 18,000 recipients. We assume a 0% attrition rate because the outcome of interest is whether a recipient applies and is, by definition, fully captured in the data.
For the pairwise comparison of 6,600 v.s. 6,600 recipients, we expect the baseline response rate to be around the lower end of the chosen range (i.e., closer to 0.25%). At the baseline response rate of 0.25%, the MDE is .27% with Bonferroni correction. We expect the baseline response rate for the pairwise comparison of 11,000 v.s. 18,000 recipients to be relatively large, at around 2%. In that scenario, the MDE is 0.52% with Bonferroni correction. As such, we are well-powered to detect economically meaningful differences in application rates.
For the applicants’ log income outcome variable, the MDE is 164% of the mean for the pairwise comparison of treatment groups with the smallest numbers of applicants, and 38% for those with the largest numbers. We take these MDE estimates as speculative, as they rest on additional assumptions about application rates. We plan to test the robustness of the effects of treatment conditions on applicant characteristics through a series of heterogeneous treatment effect estimates.