Experimental Design Details
We have assembled a list of the largest employers in the US based upon the Fortune 500. Where possible, we have split large holding companies into their constituent subsidiaries, since these entities post vacancies separately. We have selected the 100 largest firms that are feasible to audit. These firms regularly post entry-level jobs through an easily accessible online portal and are expected have vacancies in at least 125 distinct counties.
Our experiment will randomly sample 125 job vacancies for each of these firms, each in a distinct county. In cases where a firm has multiple vacancies open in a US county, we will select the job posted most recently to minimize the chances of applying to a job that has already been filled. Previous audit studies (and a pilot described below) demonstrate that sending up to 8 fictitious applications per job is feasible without employer detection (see, e.g., Banerjee et al., 2009 and Arceo-Gomez and Campos-Vasquez, 2014). Sending eight applications should enable us to precisely quantify job-level variation in discrimination within a given firm. We will stratify our analysis on race, sending four distinctively white and four distinctively black applications to each vacancy. Because our proposed design entails sampling 125 vacancies per firm, we expect to send 1,000 applications to each firm, for a total of 100,000 applications.
We have developed a software platform that constructs fictitious applicant identities. An important step in the process is the choice of racially distinctive names. We choose female and male first names from the lists in Bertrand and Mullainathan (2004). For last names we use the Decennial Census, selecting the 10 surnames with the highest racial shares among those that occur at least 10,000 times nationally for each race group, mimicking BM’s selection strategy. With BM’s nine first names for each race and gender, this provides 90 unique full names for each of the four race and gender groups. To avoid applying to the same firm with the same name more than once, we supplement this list using speeding ticket records from North Carolina. We select the most common first names with > 90% racial share among speeders ticketed between 2006 and 2018 and born between 1974 and 1979. We add 10 new names for race and gender group from this data, skipping over those already included. Fourty-seven percent of BM’s first names are encountered while doing so, suggesting that this data source captures a highly similar set of names. We combine these names with 16 last names from Census data selected according to the same criteria so that we have a total of 1000 unique full names for reach race and gender group between the two sources.
Our software generates a fictitious application specific to each job. It begins by randomly assigning a race and gender to the application, stratifying so that each job receives four white and four black applications. A race- and gender- concordant name is then assigned, sampling without replacement within a firm. The software also assigns each applicant a random date of birth that implies an age between 22 and 58 years old with uniform probability. If the vacancy requires a social security number, the software assigns one from a sample of SSNs assigned to the deceased (these SSNs are made publicly available online by the Social Security Administration).
Our fictitious applicants have phone numbers automatically provisioned by the software using a service called Twilio. Any calls to the number are sent to voicemail. Any voicemail messages received are transcribed and emailed to a central account we use to collect all callback data. Any text messages are also forwarded. The software ensures that no two applications to the same firm share a phone number. Applicants have a Gmail email address we created based on their first name, last name, and a random string of integers. There is one email address for each first name / last name combination, ensuring that the same email address is not used for multiple applications to the same firm. Applicants also have a home address that we draw randomly from data publicly available from openaddresses.io. We choose a random home address in a zip code close to the vacancy.
Each applicant graduated the year they turned 18 from a public high school drawn from database freely available from the Department of Education. We chose a school in zip codes nearby the focal job and provide the selected school’s actual street address if required. Half of the applicants also have an Associate Degree (AD). We assign AD institutions from a Department of Education database listing all degree-granting institutions in the U.S. We randomly select a two-year degree granting college in zip codes near the vacancy and a major from a list of common AD majors.
Each application also has a randomly generated employment history consisting of two to three previous jobs lasting between nine and 24 months each, with no interrupting unemployment spells. Previous employers are drawn randomly from a Reference USA database of existing firms across the country. We choose previous employers located close to the vacancy with appropriately similar job types. For example, retail job applicants have employment history at local restaurants and retailers. Each job has an appropriate position (e.g., “cashier”) and a supervisor with a name randomly chosen from a list of the most common in the U.S. Each job uses the actual address of the firm sampled from the Reference USA database and a randomly generated phone number. Each job also is assigned a set of 2-3 duties from a database of example resumes scraped from jobhero.com.
Our software supplies all application information through an online web portal. When required, the software can also generate single page PDF resumes with randomly varying fonts, spacing, section ordering, and layouts to ensure no two PDFs have identical formatting. These PDFs will be uploaded to job application portals when required.
We will submit applications by proceeding through our firm list in random order, sampling 25 establishments with 8 applications each at a time for each firm. Given that we plan to sample 125 total establishments per firm, the experiment will require five total iterations through the complete list. This strategy ensures that all firms are sampled over the course of the experiment while allowing us to concentrate our application effort on a handful of firms at a time.