Sample size: planned number of clusters
See the pre-analysis plan for a complete discussion, with footnotes. Our planned number of clusters (job openings) is at least 4,211.
A key aspect of this plan was to conduct a power analysis, based on previous studies, to determine how many observations would be necessary to detect meaningful differences in callback rates between major resume types. Based on previous studies, we saw differences of about three percentage points in the interview request rate to be possible, and we would like to be able to detect a difference of at least this magnitude between white and Indigenous applicants. Based on our calculations, we anticipated needing to apply to 4,211 jobs (8,422 applications) to detect differences in callback rates between white and Indigenous applicants of at least three percentage points.
Of course, we would ideally like to collect more data than this to be able to have a higher power, detect differences smaller than three percentage points, or to detect other mediators of discrimination (e.g., reservation upbringing, city demographics, gender, occupation) with enough precision. Our funding sources are also variable, such that we may get enough funding to go beyond this number. What we have decided is that for the main comparison of white versus Indigenous, we commit to conducting an analysis with the first 8,422 observations in addition to the resulting number of observations we ultimately get, if we exceed this amount, and discussing any differences between these two sets of results. This strikes a balance between the risk that a researcher could collect data until estimates are significant and the more salient cost of tossing out useful data that could increase precision simply to avoid this risk.