Minimum detectable effect size for main outcomes (accounting for sample
design and clustering)
For each subject (unemployment specialist) we have two observations with varying ethnicity (and constant literacy) and two observations with varying literacy (and constant ethnicity). McNemar’s test (paired binomial test) is therefore the relevant non-parametric test for our data.
Denote p11, p10, p10, p00 the sampled probabilities that a subject responds to both Czech and Roma, only Czech, only Roma, and neither of the two ethnicities, respectively. We have p11 + p10 + p01 + p00 = 1.
Let pC = p11 + p10 and pR = p11 + p01 be the overall response probabilities of receiving a response for the putative Czech and Roma senders, respectively. Finally, let \delta = pC - pR be the response differential between the two ethnicities (the discrimination effect), which after substituting yields \delta = p10 - p01.
Let n be the number of subjects (paired observations), then McNemar’s test statistic is
s = (p10 n - p01 n)^2 / (p10 n + p01 n) = (\delta^2 n) / (p10 + p01),
which under H0:=0 asymptotically follows a chi-squared distribution with one degree of freedom.
Fagerland, Lydersen, and Laake (2013) investigate Type I error frequencies and the power of alternative methods to compute the p-values. Under a wide range of parameter scenarios, the Exact unconditional McNemar test, and McNemar mid-p test, the Type I errors frequency never exceeds five percent and are almost as powerful as the asymptotic McNemar test. We, therefore, base our power calculations on the Exact unconditional McNemar test (Suissa and Shuster 1991).
In our notation, the power of the test depends on three parameters, n, \delta, p01. In our case n=457 and we consider =0.05 a substantively significant discrimination coefficient (Giulietti, Tonin, and Vlassopoulos, 2019, found four percentage points differential between whites and blacks).
In order to gauge p01, the baseline response rate in Giulietti et al. (2019) was 70 percent, setting our expectation for pC = 0.7 and implying a constraint p01 = 0.3 - p00. One now has to make a judgment about the actual size of p01. Responses to only Roma senders may happen for two main reasons: positive discrimination in favor of Roma enquirers by some subjects, and the fact that some subjects may respond to emails randomly. We believe that positive discrimination of Roma is not likely very frequent, but random responses may be. If we set p01 = 0.05 (randomness in the response occurs with the same frequency as discrimination), \delta = 0.05 implies p10 = 0.1.
The power for the one-sided Exact unconditional McNemar test with the rejection criterion \alpha = 0.05 under the stated parameters is 0.85. If we set p01 = 0.06, the corresponding power is 0.80.