Going Beyond the Mean: How Gender Affects the Distribution of Evaluations and How the Distribution of Evaluations Affects Hiring Decisions

In this project, we conduct multiple interventions to investigate the impact of various distributional metrics of job applicants on the evaluation of applicants and their likelihood of being selected for hiring. We also study whether these distributional metrics have similar effects depending on the gender of the applicants and whether the gender of the applicants is known.

2023-08-15

2024-08-15

We collect the following primary outcomes:
- Choose to hire: this is defined as the applicant, out of a pair, that is chosen by the evaluator to be recommended for hiring by the evaluator

In an earlier survey study, we found evidence that larger means and lower variance and minimums, relative to the other applicant, led to increased chance of being hired. We thus hypothesize that these patterns will follow in the experiment:

Hypothesis 1a: When mean differences are small metrics other than the mean (i.e. variance, range, maximum, minimum, outliers, skew) will have predictive power when determining which out of a pair of applicants is chosen to be hired.

Hypothesis 1b: When mean differences are small an applicant having a higher mean or maximum will increase their likelihood of being hired, while a higher variance will decrease their likelihood of being hired.

Small mean differences are defined based off of the three evaluation sets per pair that generated the smallest mean difference while retaining trade-offs between the two applicants, i.e. that one applicant did not strictly or weakly dominate the other applicant in terms of evaluations. The largest mean difference in our sample is 5.33 out of a possible range of 0-100. Because we focus on this case of smaller mean differences, we acknowledge that our results may not be generalizable to cases where the difference in means of the evaluations are more substantial.

Furthermore, based on this earlier evidence, we hypothesize the following treatment interactions:

Hypothesis 2: The predictive power of the evaluation metrics will be diminished when gender is known, compared to when gender is not known.

Hypothesis 3: When gender is known conditional on mean, we expect that metrics will have a less positive (or more negative) impact on hiring decisions for women than for men.

Hypothesis 4: The benefit (cost) from having a higher (lower) mean and a lower (higher) variance will be amplified for men (women) in mixed-gender pairs relative to when gender is not known.

To understand possible mechanisms we elicit the following secondary outcome variables:
-Time spent: as a proxy for attention, we will measure how long evaluators spend on each pair. Greater time spent will be related to greater attention spent on making the right decision.
-Gender: we will measure the likelihood that the chosen individual is female given whether gender is known and the gender composition of the pair
-Quality: we will measure the relative quality of the applicant that is chosen, as measured by their qualifications, average evaluation scores from all evaluations given, and the AI-generated scores assigned to each applicant.

See above

Our design aims to measure the impact of the distribution of applicants’ evaluations when making hiring decisions.

Not available

Randomization will be carried out by a computer.

There are multiple forms of randomization:
1) Randomization into the Gender-Known and No-Gender treatments will be across subjects. Thus, this will be at the individual level.
2) We randomize the pairs shown to each evaluator. That is what pair is shown to each evaluator take place at the individual evaluator level.
3) For each evaluator we also randomize the set of evaluation scores shown. While the evaluation sets are pre-determined what evaluation set is shown to each individual is randomized at the individual evaluator level.

No

See above, in most cases randomization occurs at the individual level, as such we cluster at the individual level.

Each evaluator is shown 3 pairs. We have 10 pairs of applicants; this means there are 30 possible combination (10*3) that could be shown to evaluators. If we collect 500 evaluators who each see 3 pairs, we can at most have 1500 possible observations, we then divide this by 2 for the gender known and gender not known treatment which gives us 750 obs. This means we have 25 observations for each possible pair combination (750/30 = 25). Total number of evaluator observations will be 1500.

Gender Known: 750
Gender Unknown: 750

Assuming the proportion of the study population that would have a value of 1 for the binary outcome in the absence of the treatment is 0.50 and conservatively treating each evaluator as an individual observation, we have a MDE of 0.11 for our main outcome of “Choose to hire”.