Experimental Design
The audit correspondence study will use a matched design, where I send, in random order, one resume and cover letter from an online degree holder to a job vacancy and a second from an in-person degree holder. I apply to vacancies in business administration across four occupation types: accounting, finance, management, and marketing. Vacancies are also spread across eight large metropolitan areas that vary by geographic region (Northeast, South, West, and Midwest) and bachelor’s degree attainment. I find vacancies on a national hiring website by searching for jobs that require a bachelor’s degree in accounting, finance, management, marketing, or a similar field, and then select a subset of vacancies appropriate for recent graduates.
Fictitious resumes will vary by college type, race, gender, work history, and skills. To ground my fictitious resumes in empirical evidence, I extract information from publicly available resumes on a job-searching website and reconstruct new resumes via a computer program that randomly selects resume characteristics (Lahey and Beasley, 2009). Resumes are anonymized (i.e., altered dates, employer names, and work descriptions) to ensure that fictitious work histories contain an assortment of jobs from real resumes but are not identifiable. Fictitious resumes are randomly assigned as one of two templates. Both templates have the same sections, including name, contact information, education, work history, and skills, but differ slightly by format, such as the font or bullet types, to avoid being suspiciously similar.
The education section lists one bachelor’s degree in business administration from a four-year public or nonprofit school. While all degrees are in a business field, the degree’s specialization align with the occupation type (e.g., applicants for finance positions will have a bachelor’s in finance or a bachelor’s in business administration with a concentration in finance). The exact names of the degrees differ slightly to represent the name of the program at the listed college but are generally equivalent. All profiles complete their bachelor’s degrees after four years of schooling and will not include a GPA.
The education section also lists high school education to convey age (see Neumark, Burn, & Button, 2019). Adult learners are typically defined as age 25 and older but that age does not represent online adult learners. The average age of an online student is 32 years old (Freidman, 2017). Assuming students graduated high school at 18 years old, I use 2009 as the average high school graduation year. To minimize the risk of detection from employers receiving two resumes with the same high school graduation year, I randomly vary one resume by plus or minus one year. Like Deming et al. (2016), I use the Common Core of Data to randomly assign public high schools from a combined statistical area to fictitious candidates with the same race as the median student. I also control for socioeconomic status by filtering out high schools in the highest and lowest quartile of free- or reduced-price lunch. Finally, I verify the selected schools have operated since 2000 and do not have other notable academic standing (e.g., selective enrollment, nationally recognized, unique pedagogies).
The experimental variation requires that each matched pair of resumes includes one online and one brick-and-mortar college. As in Deming et al. (2016), I define online colleges as those where online degrees predominate, regardless of the existence of some in-person offerings. The degree modality is not listed but is conveyed by the name recognition of predominately online colleges. To maximize the signal’s salience, I use online colleges that awarded high numbers of bachelor’s degrees in business and have operated online for at least five years.
The control resume in each pair lists a bachelor's degree from a brick-and-mortar institution near the regional labor market with little to no online offerings. Each labor market has one or two regional brick-and-mortar colleges to randomly draw from. I used fall 2019 data from the Integrated Postsecondary Education Data System (IPEDS) and the institutions' public websites to identify relevant regional institutions for each labor market. As online learning has become commonplace in postsecondary education, some of the selected brick-and-mortar institutions offer online business courses or a fully online master's degree program in business administration but do not have fully online bachelor's degree programs in business administration. In all cases, the selected brick-and-mortar institutions have very few undergraduate students enrolled exclusively online. Thus, employers are unlikely to perceive degrees from these institutions as being online. To avoid biasing results by college selectivity, the brick-and-mortar institutions are among the least selective in the region by acceptance rate.
I convey applicants’ race (white or Black) and gender (male or female) with their name. Names that convey race may also incidentally signal socioeconomic status (SES). To address this concern, I draw first names from Gaddis’s (2017) survey on the racial and SES perceptions of names from New York state birth records spanning 1994 to 2012. I select a subset of first names that are commonly perceived as white or Black (matching the race signal for at least 90 percent of survey respondents) and remove names from the first and fourth quartiles of mother’s education level (a proxy for strong SES signals). I also extract last names from Gaddis’s work, which are common surnames from 2012 Census Data with a population-level racial occurrence of at least 50% white or Black. I select four first names for each gender and race combination and four last names per racial group. Rather than using all sixty-four combinations of names, I take Bennett’s (2022) approach and select four full names for each gender and race combination.
Realistic resumes must also include work histories. I will fill the time between high school graduation (2009) and college enrollment (2019) with ten years of work experience. Since it is common for online students to work during their studies (Friedman, 2017), I will include an additional four years of work experience throughout college enrollment. The applicants’ long work histories are what set apart this audit from similar experiments about online degrees. To construct work histories, I extract job titles and descriptions from a sample of publicly available Indeed resumes. I screen for high-quality resumes (e.g., complete descriptions of job responsibilities) of online degree holders with prior work experience in each occupation type. The job descriptions are rephrased, anonymized (e.g., changing the employer’s name), and deconstructed as separate job templates. To ensure work histories look similar to adult learners on Indeed, the first work experience following high school will not directly align with the occupation type (e.g., cashier). I create four complete work history templates for each of the four occupation types. Work history templates are tailored to each metropolitan area by substituting the employers’ names with local employers or chain businesses that operate in the region.
For each occupation type, I generate relevant skill templates (technical and soft) with the same approach. The addition of skills provides a useful way to address Heckman’s critique of experimental labor discrimination studies. Heckman argues that differences in the variance of unobservable characteristics related to productivity can generate bias (Heckman, 1998). However, there is no evidence that online students have a higher variance of unobservable characteristics than in-person students. For the sake of robustness, I address the Heckman critique by following the approach used in Neumark, Burn, & Button (2019), and randomly assign additional skills to convey variations in human capital investment. I control for high skill resumes in the regression analysis.
The final resume elements to include are applicants’ address and contact information. Applicants live in the same metropolitan area as the vacancy to which they are applying. I select a subset of the fictitious addresses used in Neumark et al. (2019). To provide employers with several options to callback, I create Gmail accounts for each name combination and phone numbers for each metropolitan area and college type combination. Employers can also contact applicants through their Indeed profile.
Job vacancies that require a bachelor’s degree typically ask for, or at least allow, the submission of a cover letter. With each resume, I send a cover letter randomly assigned as one of two templates. The second fictitious applicant in the matched pair uses the other template. Like resumes, the cover letter templates slightly differ in formatting and phrasing but concisely address the same points, including expressing interest in the vacancy, a description of the candidate’s professional experience, and an explanation of the technical and soft skills the candidate would bring to the role. Time constraints prevent me from tailoring cover letters to individual job postings; however, they are tailored by occupation type to emphasize relevant professional experience and skills (e.g., accounting candidates have strong organizational skills and experience managing financial records).
Full data collection will begin in December 2023.