AI or human? Applicants’ decisions in discrimination settings

The laboratory experiment investigates whether the potential of discrimination influences applicants’ preferences for a hiring algorithm.
To this end, applicants can choose to be hired by a human manager or by an artificial intelligence in different hiring situations which entail the potential of discrimination.

2023-11-23

2023-12-31

The primary outcome variable is the applicant’s decision on whether the hiring algorithm or the human manager takes their hiring decision.

The main analysis will test whether the choices for the decision makers differ depending on the potential of discrimination.

Secondary outcomes are the applicants’ beliefs about the decision makers’ choices as well as their beliefs about the performances in the treatment groups. These outcomes give insights on the perceived potential of discrimination.

The experiment will be conducted via Prolific.

We conduct an online experiment based on the design of Dargnies et al. (2022). In the experiment, participants are assigned either the role of applicants or managers. Applicants perform two work sample tasks and one job task. Managers and the algorithm receive the work sample task performance of a worker pair of similar work sample task performance and decide whom to hire based on this. Before the hiring decision, both the manager and the algorithm receive a random subset of workers’ work sample and job task performance data as a training set. Applicants are informed about the training opportunity and how the algorithm and the manager take the hiring decision. Applicants are asked to decide whether a human manager or an artificial intelligence should take their hiring decision.
The novelty in our study is the stimulation of anticipated discrimination. Following standard approaches to labor market discrimination, we distinguish between taste-based and statistical discrimination. The perceived potential for discrimination is achieved using minimal groups based on colors as well as training opportunities for applicants.

Further survey evidence as well as beliefs about other participants’ actions and performances are elicited at the end of the experiment.

The experiment will be conducted via Prolific.

The final sample excludes subjects that:
- Quit the experiment before its end
- Fail both attention checks

The experiment is designed as follows:
We conduct an online experiment based on the design of Dargnies et al. (2022). In the experiment, participants are assigned either the role of applicants or managers. A between subject design is used. Applicants perform two work sample tasks and one job task. All tasks are incentivized and require real effort. Managers and the algorithm receive the work sample task performance of a worker pair of similar performance and decide whom to hire based on this. Before the hiring decision, both the manager and the algorithm receive a random subset of 100 workers’ work sample and job task performance data as a training set. We inform the applicants that managers and the algorithm receive the same information. Applicants are asked to decide whether a human manager or an artificial intelligence should take their hiring decision. They receive a bonus payment if they are hired by the chosen decision maker. Managers take several hiring decisions which are highly incentivized.
The novelty in our study is the stimulation of anticipated discrimination. Following standard approaches to labor market discrimination, we distinguish between taste-based and statistical discrimination. The baseline condition of the experiment does not incorporate discrimination and is intended to capture general algorithm aversion. The taste-based discrimination treatment differs from the baseline by assigning applicants and managers to one of two groups based on colors (green, purple). Applicants thus face a hiring decision made by either an in-group or out-group manager. In the statistical discrimination treatment, applicants are assigned to one of two groups (training, non-training). 70% of the members of the training group solve the job task twice. Consequently, the training group is likely to have a higher average performance on the final job task. In this treatment, managers are not assigned to groups.
Immediately after choosing who should take their hiring decision, applicants are asked to state the reasons for their choice as answer to an open-ended question (not incentivized). At the end of the experiment, applicants state their beliefs about how many times the decision makers choose the better-performing applicant. Moreover, they estimate their own relative performance. Applicants in the discrimination treatments state their beliefs about the decision makers’ choices with respect to the group affiliation of the applicants (in-group/training group). In addition, they are asked to evaluate the performance differences between the groups in the treatment conditions. All those questions are incentivized. In all conditions, we elicit algorithm aversion using the Attitude Towards Artificial Intelligence (ATAI) scale of Sindermann et al. (2021), experience with algorithms in hiring, discrimination in hiring contexts, discrimination in the daily life using the short version of the Everyday Discrimination scale of Sternthal et al. (2011), as well as demographic information.

Applicants’ and managers’ sessions are run separately.

The assignment to roles (applicant, manager) as well as to the treatments (baseline, taste-based discrimination, statistical discrimination) is done in Prolific.
Assignment to the groups (in- or outgroup, training or non-training group) within the treatments will be done in oTree.

Individual

No

The number of clusters is the same as the number of observations.

1350 participants on Prolific

We plan to have 1250 applicants and 100 managers.

From the 1250 applicants, 250 are assigned to the baseline condition. 500 belong to the taste-based discrimination treatment in which 250 belong to the green group and 250 to the purple group. 500 participants are assigned to the statistical discrimination treatment in which 250 belong to the training group and 250 to the non-training group.