Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
We use existing apartment listing data from the same online platform in a pre-trial in Houston, TX to identify the sample size requirements for statistical power. The Houston pre-trial data contains 1563 listings. The pre-trial yielded a 17.9% response rate to white names and 16.7% to names associated with African American or LatinX/Hispanic names (non white names). It also yielded a relatively balanced sample with respect to proximity to TRI facilities: 45% of the rental properties where in the neighborhood of a toxic plant (within 1 mile).
To compute the sample sizes and the minimum detectable effects of the interaction of race and proximity to toxic plant we assume 90% test power and .05 significance level. In the small of data from the Houston pre-trial, we estimate an odds ratio of 1.27 (0.42) for the interaction. Standard errors are clustered at the Houston zip code level. We then simulate the effect of increasing sample size in a conditional logit model with paired inquiries. Simulation results suggest an effect size 1.54 that can be detected with 3017 properties. Figures 3 and 4 in our supporting materials shows simulation results for different sample sizes, for odds ration and p-values. Alternatively, if we use Demidenko (2007, 2008) approach to calculate the number of listings it yields that we need about 2,433 properties to obtain for that detectable odds ratio.
Phillips (2016) provides evidence of within-trial impacts when multiple inquiries sent in matched correspondence designs in competitive labor markets. In a sample restricted to responses to the first inquiry and based on a simple logit model, our simulations show that we should be able to detect an effect with an odds ratio of 1.43 at 3676 properties. Figures 5 and 6 shows the results of these simulations.
These power calculations are limited by available data from the Houston pre-trial and the incidence of discriminatory behavior that may be particular to the Houston housing market.
Demidenko E. (2007). "Sample size determination for logistic regression revisited." Statistics in Medicine 26:3385-3397
Demidenko E. (2008) "Sample size and optimal design for logistic regression with binary interaction." Statistics in Medicine, 27:36-46
Phillips, David C. "Do comparisons of fictional applicants measure discrimination when search externalities are present? evidence from existing experiments." The Economic Journal (2016).