Wiseguys or Wise Guys? How do high school equivalency credentials affect third-party appraisals of job applicants' resumes?

We will recruit roughly 1,000 evaluators with managerial experience from five states (CA, FL, GA, NY, and TX) and ask them to evaluate resumes of applicants with randomly assigned characteristics, including implied gender, implied race/ethnicity, work experience, volunteer experience, and educational attainment. We will also randomly assign the entry-level job description that is presented to evaluators between a customer service position with relatively high social skill task content versus an administrative job with relatively low social skill task content and test whether the impact of educational credentials is related to job characteristics.

Before collecting data from all 1,000 evaluators, we will run a pilot of 50 evaluators to test survey functionality. Once we confirm that they survey works as intended, we will scale up data collection to the remaining 950 evaluators.

We will assess the impact of observing different resume characteristics on these hiring managers' appraisals of the resumes.

Our primary analysis will focus on assessing the relationship between observing a high school equivalency credential and evaluator appraisals relative to applications with 1) no educational credential reported, 2) a traditional high school diploma, and 3) some college.

Our exploratory analyses will assess whether there are differential impacts of listing a high school equivalency credential on evaluator rating for the subset of applicants who report having a high school equivalency credential and also report earning a "GED College Readiness" designation in math or language arts as well as assessing whether the impact of educational credentials varies by job type (customer service vs. data entry). We will also examine whether different resume characteristics moderate the effect of educational credentials on evaluations (e.g., more or less experience; racial or gender differences in impacts (as well as their interaction); type of experience [volunteer vs. paid work; sector of paid work], etc.). We will present our primary analyses pooled across cities, but also publish results separately by state. Additionally, we will assess whether earning a HS diploma from a high-performing vs. low-performing high school moderates any gap in evaluation between HSE equivalency credential holders and traditional HS grads, with and without limiting our sample to evaluators who accurately identify at least 2 of 3 high performing high schools (or 2 of 3 low-performing high schools) in their metro area. Finally, we will assess the mechanisms that explain why hiring managers reward or penalize applicants with HSE by examining whether participants ratings of applicants or preferences for applicants with HSE credentials varies with the factors they report considering as they evaluated candidates as well as the factors they state are "most important" for hiring managers to consider (e.g., social skills, problem solving skills, language skills, alignment of work experience with job, criminal history, etc.).

2025-03-12

2025-04-15

1) Overall evaluator scores of resumes. Evaluators will be asked to rate each candidate overall on a scale of 1-10. These scores will be standardized to have a mean of zero and a standard deviation of one. We will assess the impact of resume characteristics on primary outcome 2 using ordinary least squares regression with evaluator fixed effects, clustering our standard errors at the evaluator level.
2) The probability that an Evaluator selects a resume with a given realized value of a characteristic, conditioning on the realized values of all other resume characteristics. We will assess the impact of resume characteristics on primary outcome 2 using logistic regression with evaluator fixed effects, clustering our standard errors at the evaluator level.

Overall evaluator scores will be derived by summing five sub ratings (overall quality; critical thinking & problem-solving; grit & persistence; relevant training, and relevant experience). These scores will then be standardized to have a mean of zero and a standard deviation of one, ensuring comparability across different resumes and raters.

Selection probability will be constructed from a binary indicator for each resume, where a "1" indicates the resume was chosen as the preferred candidate in a pairwise comparison and "0" indicates it was not. This will measure the likelihood of a resume being selected based on its characteristics, including educational background.

Evaluator sub scores of resumes: Evaluators will rate each candidate on specific dimensions such as critical thinking & problem-solving and grit & persistence. These ratings will be standardized to have a mean of zero and a standard deviation of one, ensuring uniformity in how the sub scores are compared across all resumes evaluated. We will also examine the impact of reporting a HSE credential on each component of the primary rating index and use these subscores to test whether reporting an HSE credential influences evaluators’ appraisal of candidates’ critical thinking and problem solving skills (subindex 1: academic skills) differently than their appraisal of candidates’ grit and persistence (subindex 2: non-academic skills) as predicted by Araujo, Gottlieb, and Moreira (2007).

The experimental design involves recruiting 1,000 hiring managers from five states (CA, FL, GA, NY, TX) to evaluate resumes with randomly assigned characteristics. Each resume will have variations in implied gender, implied race/ethnicity, work experience, volunteer experience, and educational attainment (GED, high school diploma, no education, or some college). Additionally, job descriptions will be randomly assigned to either a customer service role (with high social skill content) or an administrative role (with low social skill content). Hiring managers will rate each resume on various attributes and choose a preferred candidate from pairs of resumes, allowing us to assess the impact of educational credentials on their evaluations. A subset of applicants who report having a high school equivalency credential will also report earning a "GED College Readiness" designation in math or language arts.

The randomization of the order in which job types are presented to evaluators is implemented using the randomizer function in Qualtrics.

The randomization of which resume pairs are presented to evaluators (and their order) is implemented using the "Loop and Merge" function in Qualtrics where 1800 candidate pairs are randomly sorted and divided evenly among modules (180 per module), and evaluators are presented with 1 pair out of these 180 options for each of 10 comparisons.

The randomization of characteristics to resumes is done in office by a computer to assign educational credentials and job descriptions to the resumes (e.g., by assigning each observation a uniform random number and assigning a particular characteristic in proportion to the number of realized values of that characteristic). This ensures that each characteristic is randomly allocated, minimizing bias and allowing for the unbiased assessment of the impact of educational credentials on evaluator appraisals.

The unit of randomization is the individual resume. Each resume will have randomly assigned characteristics, such as implied gender, implied race/ethnicity, work experience, volunteer experience, and educational attainment (GED, high school diploma, no education, or some college). Evaluators will be randomly assigned pairs of resumes to assess, allowing for a within-subject, individual-level randomization.

Yes

This is a within-subject experiment, so the randomization is clustered within 1000 respondents from 5 cities. Our primary results will include an evaluator fixed effect to account for idiosyncratic differences in evaluators' average scores, with standard errors clustered by evaluator. Evaluators in different states will rate different pools of applications (e.g., Texas evaluators will be presented with a hypothetical candidate applying for a job in Houston, Texas while California evaluators will be presented with a hypothetical candidate applying for jobs in Los Angeles, CA). Each hypothetical job search scenario is constructed around a candidate applying for a customer service or data entry job in that state's largest metro area (CA: Los Angeles; FL: Miami; GA: Atlanta; NY: New York City; TX: Houston). All analyses will also include a job type fixed effect.

The planned number of observations is 20,000 resumes (1,000 evaluators × 20 resumes each).

1,000 evaluators evaluating 20 resumes each (20,000 resume ratings/selections) for primary outcomes 1 and 2.

We use the power twomeans command in stata to estimate the Minimum Detectable Effect Size (MDES) at 80% power for our standardized outcomes, assuming an intra-cluster correlation of 0.2 and reducing the size of the test from alpha=0.05 to alpha=0.025 to account for our multiple primary outcomes: power twomeans 0, power(.8) sd(1) alpha(0.025) m1(10) m2(10) k1(1000) k2(1000) rho(0.2) MDES=0.0730 standard deviations Similarly assuming the mean of our binary outcome 1(ResumeChosen) is 0.5 by construction, we can use a similar process to estimate the MDES in percentage points for our second primary outcomes: power twomeans 0, power(.8) sd(0.5) alpha(0.025) m1(10) m2(10) k1(1000) k2(1000) rho(0.2) MDES=3.65pp