Inference Effects in Discrimination Measurement

Participants take part in an online recruitment-style evaluation task. They are asked to assess a candidate who has completed a logical reasoning problem. Before making their assessment, participants are randomly assigned to receive one of several versions of the evaluation context. These versions vary the background information provided about the recruitment process and selected features of the candidate-evaluation scenario. Participants then provide ratings of the candidate’s performance based on the information shown to them.

Participants take part in an online survey experiment framed as a recruitment-style evaluation task, and are informed that such tasks are commonly used in financial sector recruitment. The study uses a 2 × 2 × 2 between-subjects design. Participants are randomly assigned to one of two information conditions, one of two prior-evaluator conditions, and one of two candidate-gender conditions.

In the information manipulation, participants are assigned either to a control condition that provides general background information about progress and consistency in financial-sector evaluation, or to an information condition that additionally presents evidence that gender can affect how performance is judged in this sector.

In the prior-evaluator manipulation, the candidate’s answer to a logical reasoning problem is described as having been previously assessed either by a human investment banking professional or by an algorithmic evaluator. In the candidate-gender manipulation, participants evaluate either a female sounding name or a male candidate.

Participants first review background information about the recruitment context. They then see a candidate profile and the candidate’s answer to a logical reasoning problem. Participants rate the candidate’s performance and underlying ability based on the candidate’s answer. They subsequently receive additional information about the candidate’s reasoning and provide a second assessment. The design allows the study to examine how information about the prior evaluation process affects judgments of candidate quality when evaluators observe limited versus richer performance information.

2026-07-09

2026-07-24

The first primary outcome is the participant’s estimate of the probability that the candidate’s answer to the logical reasoning problem is correct, after observing the candidate’s answer but before observing the candidate’s reasoning.

The second primary outcome is the participant’s estimate of the probability that the candidate’s answer is correct after observing both the candidate’s answer and the candidate’s reasoning.

This study is an online survey experiment in which participants complete a recruitment-style evaluation task. Participants are asked to review information about a job candidate who has completed a logical reasoning problem. They are randomly assigned to different versions of the evaluation context and candidate-evaluation scenario.

Participants first read background information about the recruitment setting. They then see a candidate profile and the candidate’s answer to a logical reasoning problem, and provide an initial assessment of the candidate’s performance. Participants subsequently receive additional information about the candidate’s reasoning and provide a second assessment. The study is designed to examine how information about the evaluation process affects judgments of candidate quality when participants observe different amounts of performance information.

This study is an online survey experiment framed as a recruitment-style evaluation task. Participants are informed that logical reasoning tasks are commonly used in financial-sector recruitment. The experiment uses a 2 × 2 × 2 between-subjects design. Participants are randomly assigned to one of two information conditions, one of two prior-evaluator conditions, and one of two candidate-gender conditions.

First, participants are randomly assigned to an information condition. In the control condition, participants receive general background information about progress and consistency in financial-sector evaluation. In the information condition, participants receive the same general background information plus additional evidence suggesting that gender can affect how performance is judged in this sector.

Second, participants are randomly assigned to a prior-evaluator condition. The candidate’s answer to a logical reasoning problem is described as having been previously assessed either by a human investment-banking professional or by an algorithmic evaluator.

Third, participants are randomly assigned to a candidate-gender condition. The candidate is presented using either a female-sounding name or a male-sounding name.

After reading the background information, participants see a candidate profile and the candidate’s answer to a logical reasoning problem. They then provide an initial assessment of the candidate’s performance by estimating the probability that the candidate’s answer is correct. Participants subsequently receive additional information about the candidate’s reasoning and provide a second assessment of the probability that the candidate’s answer is correct.

This design allows the study to examine how information about the prior evaluation process affects judgments of candidate quality, and whether these judgments differ when participants observe limited versus richer performance information.

Randomization is implemented in Qualtrics using the platform’s built-in randomizer. Participants are individually randomized to one of the experimental conditions with equal assignment probabilities across conditions.

The unit of randomization is the individual participant. All experimental conditions are assigned at the individual level, and there is no cluster-level randomization.

No

1,280 individual participants

1,280 individual participants

The study has eight experimental arms, with approximately 160 individual participants assigned to each arm:

160 participants: Control information × human investment-banking evaluator × female-sounding candidate name

160 participants: Control information × human investment-banking evaluator × male-sounding candidate name

160 participants: Control information × algorithmic evaluator × female-sounding candidate name

160 participants: Control information × algorithmic evaluator × male-sounding candidate name

160 participants: Information treatment × human investment-banking evaluator × female-sounding candidate name

160 participants: Information treatment × human investment-banking evaluator × male-sounding candidate name

160 participants: Information treatment × algorithmic evaluator × female-sounding candidate name

160 participants: Information treatment × algorithmic evaluator × male-sounding candidate name

The planned sample size is 1,280 individual participants, with approximately 160 participants in each of the eight experimental cells. The outcome variables are measured as probability estimates on a 0–100 scale, so the unit of the minimum detectable effect is percentage points on this scale. Assuming 80% power, a two-sided 5% significance level, no clustering, equal cell sizes, and an outcome standard deviation of 20 points, the minimum detectable effect for a simple comparison between two equally sized experimental cells is approximately 0.31 standard deviations, or 6.2 percentage points. Because the main estimands involve interaction effects, the relevant minimum detectable effects are larger. Under the same assumptions, the approximate minimum detectable effect for the 2 × 2 candidate-gender-by-evaluator interaction within one information condition is approximately 0.44 standard deviations, or 8.8 percentage points. The approximate minimum detectable effect for the full information-by-candidate-gender-by-evaluator interaction is approximately 0.63 standard deviations, or 12.6 percentage points. These calculations are benchmarks; realized power will depend on observed outcome variance, realized cell balance, and the final analysis specification.