Experimental Design
Phase 1: We will use data from Stanton (2023), including ~1,000 applications, which included both cisgender and transgender White applicants in four markets
Phase 2: For this correspondence study, we will submit 5000 applications to 2500 job postings with unique employers on Craigslist. The correspondence study methodology means that all respondent characteristics will be held constant or randomized between resumes. As a result, we will be able to attribute any differences in the likelihood of receiving a callback to the employer perception of the traits (gender identity and race) made salient to the employer. This enables causal identification of the impact of gender identity and race on callback rates, which themselves are a strong proxy for employer demand.
We signal whether the applicant is a man or woman by randomly selecting first names from 2003 SSA data on newborn first names that are at least 90% female or 90% male.
We signal transgender identity by including a statement near the top of each resume stating that the applicant’s preferred and legal names differ. For example, “My preferred name is John, but my legal name is Mary.” One name is unambiguously masculine, while the other name is unambiguously feminine to establish a contrast in gender between the applicant’s legal name and preferred name that signals transgender identity.
We signal non-binary identity by putting they/them pronouns next to names that are randomly selected to be non-binary. The first name is either one of the randomly selected male or female names or a gender ambiguous first name. We randomly select gender ambiguous first names from 2003 SSA data on newborn first names by eliminating first names that do not have a female to male ratio of 40-60% and that are less than 85% white. There are not gender ambiguous names that effectively signal black identity.
We signal race by randomly selecting first and last names, which are used predominantly by either white or black individuals. The first names are from 2003 SSA data on newborn first names that are at least 90% white or at least 50% black. The surnames are from the 2000 US Census that are at least 90% white or at least 40% black. While there are many first and last names that are at least 90% white, there are few that are above 40% black, which is why the cutoff percentage is lower for black first and last names.
Market selection
Craigslist operates in 418 geographically distinct markets within the United States. However, some markets are not used frequently enough by employers to provide sufficient unique job postings for this study.
We will include the four markets from Phase 1: Houston, Los Angeles, New York, and Phoenix.
We will screen out markets that do not have sufficient posting activity by documenting the number of postings in the key sectors (food and/or retail) in a specific window for each market and removing those with limited activity. This may also exclude some states from the study.
To select states from the remainder, we will generate a ranked list of states by vote shares in the 2020 presidential election. We will create paired strata (the first and second most Republican, the third and fourth, etc.) and randomly select one state from each strata to include in the study, yielding approximately 24 states.
In states with more than 2 markets, we will then randomly select two per state, with the probability of sampling proportional to population.
Developing Resumes
Each resume will be for the food or retail industry, with two job experiences in said industry (one customer-facing, one non-customer-facing). Job descriptions will be generated using ChatGPT and then edited appropriately.
The jobs and companies will be standardized. Then, the locations of these jobs will be selected from the companies’ locations near the applicants’ cities of residence.
Each resume will have a high school degree from the specific city of residence.
Identifying Job Postings
Because of the large scale of this field experiment, we will use python script to scrape job postings across the selected markets, using only the food service and retail categories.
Included postings will be screened to ensure they are (1) entry-level and require no more than a high school diploma; (2) require only a resume to apply; and (3) provide an email address to apply. We will also screen data to ensure there is at most one post per employer.
Postings will be updated at regular intervals during the data collection period, pulling only postings from the past week. If there are more eligible postings than the sample target, positions will be randomly selected.
Submitting Resumes
We will randomize gender identity (cisgender man, cisgender woman, transgender man, transgender woman, or non-binary) and race (black or white) for the resumes.
Based on the randomization of applicant gender identity and race, the research assistants will fill the resume templates with the appropriate applicant information (name, signal or transgender or non-binary identity, phone number, and email address).
Research assistants will submit the 5000 applications to 2500 job postings, which we anticipate will take approximately two months.
Data Collection
We will track callbacks (employer responses to an application offering an interview or job to the applicant) by regularly monitoring the applicant phone numbers and email accounts.
We will organize the data to measure the total number of callbacks for each cell to compare the callback rates and rates of hiring discrimination.
We will decline the callback or interview offer with a standardized message writing that the applicant has already accepted another position and thanking the employer for their time.