Back to History Current Version

The Impacts of Soft Affirmative Action: Experimental Evidence

Last registered on November 02, 2021

Pre-Trial

Trial Information

General Information

Title
Could affirmative action backfire?
RCT ID
AEARCTR-0007383
Initial registration date
March 18, 2021

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
March 22, 2021, 1:17 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
November 02, 2021, 1:11 AM EDT

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Region

Primary Investigator

Affiliation
Queensland University of Technology

Other Primary Investigator(s)

PI Affiliation
Queensland University of Technology

Additional Trial Information

Status
In development
Start date
2021-11-01
End date
2024-01-31
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
The aim of affirmative action (AA) policies is to increase the representation of minorities in candidate pools for hiring and/or promotions. In this study, we plan to use the controlled setting of a lab experiment to find evidence and understand the true size and nature of the spillover effects of a soft AA policy on employer discrimination. It allows us to determine 1) whether this effect is predominately positive or negative, and 2) whether it is primarily driven by behavioural preferences (taste-based discrimination) or rational choice (statistical discrimination). We do this by separating hiring decisions from output estimation decisions, and by comparing AA policies for an ethnic minority group with for a random “priority” group that has no distinct characteristics. Our findings aim to provide evidence and insights into the mechanisms of spillover effects of soft AA policies in the labour market.
External Link(s)

Registration Citation

Citation
Hu, Hairong and Gregory Kubitz. 2021. "Could affirmative action backfire?." AEA RCT Registry. November 02. https://doi.org/10.1257/rct.7383-2.4000000000000004
Experimental Details

Interventions

Intervention(s)
1) Baseline: No affirmative action for the minority group in selecting the candidate pool.
2) Baseline – Type (2): No affirmative action for the minority group in selecting the candidate pool. No ethnic type informed.
3) Treatment (Intervention) - AA policy - minority (3): There is a soft AA for minorities in selecting the candidate pool. Ethnic type informed.
4) Controlled- AA policy – Lucky (4): There is a soft AA for a random group (to have priority) in selecting the candidate pool. No ethnic type informed but the priority status will be informed.


Intervention Start Date
2021-12-01
Intervention End Date
2024-01-01

Primary Outcomes

Primary Outcomes (end points)
Hiring decisions: 1) percentage % of minorities candidates being hired 2) probability of a minority candidate being hired, controlled by ethic type, scores(rank), age(different from the employer's age). We compare these two outcome variables in different treatments.

Estimation decisions: 1) mean estimated scores for majority candidates and minorities candidates in different treatments 2) Multiplier, measured by (estimated scores/given scores) -1. We will compare the average value of the multiplier among the majorities candidates and minorities candidates. 3) For each treatment, we will run a linear regression on estimated scores_it (i for an individual candidate, t for the session) = a0+a1*minority_it +b1*scores_it +b2*scores_it*minority_it (b2*scores_it*lucky_it in treatment (4))+b3*age+eit. b2 captures the ethnic/priority difference in the impact of given scores on estimated scores (signal effects). Therefore, we can capture the signal effects via b2. If b2 is negative and different from 0 under AA policy minority (3) and AA policy lucky(4), the signal effect is significant, and the positive productivity signals of a minority/affirmed candidate are less effective than that of a majority/unaffirmed candidate.

Primary Outcomes (explanation)
- Firstly, we need to test whether the employers hold a natural bias against the ethnic minority because of a stereotype that an ethnic minority is likely to perform worse than the majority in the word anagram task. We will compare Baseline(1) and Baseline type (2) to test the bias. If we find that both the probability of minority candidates being hired and the average estimated scores of minority candidates are lower in the baseline type (2), the natural bias against minorities exist. This bias is the prior condition for a possible exposure effect, which we will explain in the next paragraph.

- Secondly, we plan to detect four effects through the outcome variables in Soft AA minority(3) vs. Baseline type(2).
1) Hiring decisions:
1) (positive) Frequency effects: The introduction of AA policy for minorities will increase the proportion of minority candidates in the pool (>=50%). This is likely to increase the likelihood of a minority to be hired, including the percentage % of minorities candidates is higher than % of majorities candidates, and the probability of a minority candidate being hired is greater than a majority candidate under soft AA minority(3), holding all else equal.
2) (negative) Unfairness effects: The introduction of AA policies may be accompanied by a perceived procedural unfairness to the affirmed group and give employers a greater preference for the unaffirmed group. This is likely to reduce the likelihood of a minority being hired, and therefore the percentage of minority candidates is lower than the percentage of majority candidates under the soft AA minority (3) if we hold other things equal.

3) (positive) Exposure effects: Through comparing Baseline(1) and Baseline Type (2) in hiring decisions (see the first paragraph), we can know whether majority employers hold a natural bias against minority employees during hiring decisions. If the natural bias exists, the introduction of AA policy for minorities can help employers have greater exposure to minorities, and understand there is no ability difference between majorities and minorities candidates. This is likely to eliminate the natural bias. The percentage % of minorities candidates should be no different from majorities candidates, and the probability of a minority candidate being hired should be the same as that of a majority candidate under soft AA minority(3). If the natural bias does not exist, the exposure effect does not exist and we do not need to discuss it.

2) Estimation decisions:
1) (positive) Exposure effects: The exposure effects can also help employers overcome the stereotype and realise there is no ability difference in the task. We expect to see there are no differences in mean estimated scores in the Soft AA minority (3), but there are positive differences in the Baseline type (2), holding everything equal.
2) (negative) Signal effects: The introduction of a soft AA policy would weaken the positive productivity signals of the affirmed group because some candidates from the affirmed group would not pass the pre-screen process without the soft AA policy. This will cause employers to estimate lower scores for the affirmed candidates than the unaffirmed candidates if employers observe the same given scores. Signal effects should be the same in both soft AA minority(3) and soft AA lucky(4) because it is the backfire effect caused by AA policy itself. We expect to see 1) mean estimates scores of minority/affirmed group < mean estimated scores of majority/unaffirmed group under the soft AA minority(3) and soft AA lucky(4); 2) In soft AA minority (3) & soft AA lucky (4), 0<the mean multiplier for minority/affirmed group<the mean multiplier for majority/unaffirmed (given the positive signals). 3) b2 (the coefficient of interaction term of minority/affirmed group * score) should be significant and negative under both soft AA minority(3) and soft AA lucky(4), indicating that signal effects do exist, and the positive productivity signals of a minority/an affirmed candidate are less effective than that of a majority/an unaffirmed candidate after the introduction of a soft AA policy.

Secondary Outcomes

Secondary Outcomes (end points)
1) Hypothesis 1: Behaviour story (Large unfairness impact – hired less but estimated same).
- In the hiring decision: The negative spillover effect is much larger in the soft AA policy minority(3) than in the soft AA policy lucky(4).
- In the estimation decision: Expect no difference in soft AA policy minority(3) and in the soft AA policy lucky(4).

2) Hypothesis 2: Rational story (Small unfairness impact – hired more or indifferent but estimated less) within Soft AA minority(3).
- In the hiring decision: Exposure and frequency effects (positive spillover) dominate the signal effects (negative spillover).
- In the estimation decision: Signal effects (negative spillover) dominate the exposure effects (positive spillover).




Secondary Outcomes (explanation)
In the secondary outcomes, we will use the difference-in-difference method to compare the differences in soft AA minority(3) and baseline type(2), and the differences in soft AA lucky(3) and baseline(1). This is to determine the differential effects between a AA policy for a minority group and a AA policy for a random group, and therefore understand whether a AA policy itself has negative spillover effects.

We have two possible outcomes:
1) Hypothesis 1: Behaviour story: The hiring decisions are not consistent with estimation. We expect there is a large unfairness impact - employers hire fewer minorities but estimated the same. The differences of soft AA minority(3) - baseline type(2) > The differences of soft AA lucky(3) - baseline(1) in hiring decision.
- In the hiring decision: The negative spillover effect is much larger in the soft AA policy minority(3) than in the soft AA policy lucky(4). We expect the likelihood of a minority being hired is much lower than that of a majority because unfairness effects dominate all other effects (other effects are negligible), and unfairness only exists in soft AA policy lucky, if unfairness is stronger with the out-group than with a random group.
- In the estimation decision: Expect no difference in soft AA policy minority(3) and in the soft AA policy lucky(4). This is because the signal effects and exposure effects are negligible.

2) Hypothesis 2: Rational story: The hiring decisions are consistent with estimation. We expect there is a negligible unfairness impact - employers will hire indifferent between two groups but estimated less for the minority/affirmed group within soft AA minority(3) and soft AA lucky(4). The differences of soft AA minority(3) - baseline type(2) = The differences of soft AA lucky(3) - baseline(1) in estimation decision.
- In the hiring decision: Exposure and frequency effects dominate employers' hiring decision and we expect the likelihood of a minority/affirmed group is slightly higher (because a AA policy brings higher frequency) or indifferent with the likelihood of a majority (because a AA policy eliminates the natural bias).
- In the estimation decision: Signal effects (negative spillover) dominate the exposure effects (positive spillover). And we expect 1) mean estimated scores of minorities/affirmed group < mean estimated scores of majorities/unaffirmed group 2) Multipliers of minorities/affirmed group < Multipliers of majorities/unaffirmed group 3) b2 is negative and significant in both soft AA minority(3) and soft AA lucky(4). The value of b2 should has no difference under these two treatments.

Experimental Design

Experimental Design
In this experiment, we have two phases: 1) preliminary phase, in which we aim to recruit 100 participants to complete a series of tasks. 2) The secondary phase, in which we aim to recruit 100 participants for every four treatments to complete a hiring game with a hiring decision and 4 estimating decisions.

The preliminary phase is designed to generate actual profiles of candidates for use in the second phase of the hiring game. The benefit of using actual profiles is to introduce actual costs for discriminatory behaviour and therefore capture the actual level of employer discrimination (Hedegaard & Tyran, 2018). During this phase, we are going to ask participants to finish an individual experiment, including five individual tasks with 2 minutes each. The individual task is a 4-letter word anagram that participants need to correctly rearrange as many as possible sets of 4 letters to a meaningful word in 2 minutes. At the end of this phase, we will ask participants to finish an exit survey to capture their individual demographic differences. With five 4-letter anagram tasks, we were able to measure each individual's productivity and generate the inputs for the second phase of the hiring game.

The second phase is a hiring game, in which we will introduce four different treatments, a soft AA policy for an ethnic minority group, a soft AA policy for a randomly selected group, and baselines both with and without information about ethnicity. In the second part of a hiring game, we will only recruit the majority as our participants. The majority are those who self-reported as White, currently live in the U.S., are born in the U.S., use English as their first native language. And all the participants for this experiment need to make two decisions: 1) hiring decision 2) estimation decision.

Prior to making a hiring decision, all profiles will go through a “pre-screen process" in which the computer will randomly select one of the five tasks completed by the individuals of profiles during the preliminary phase and rank all 12 profiles. Only 4 profiles will be selected as candidates during the hiring decision. During the hiring decision, participants will receive the profiles of four candidates, including scores of another drawn task (different from task used in “pre-screen process"), and age. Whether the ethnicity type of each candidate is included in the profiles, and the way for selecting 4 candidates' profiles are different treatments by treatments. We have 4 different treatments.

The preliminary phase is designed to generate actual profiles of candidates for use in the second phase of the hiring game. The benefit of using actual profiles is to introduce actual costs for discriminatory behaviour and therefore capture the actual level of employer discrimination (Hedegaard & Tyran, 2018).

The second phase is a hiring game, in which we will introduce four different treatments, a soft AA policy for an ethnic minority group, a soft AA policy for a randomly selected group, and baselines both with and without information about ethnicity.

Prior to making a hiring decision, all profiles will go through a “pre-screen process" in which the computer will randomly select one of the five tasks completed by the individuals of profiles during the preliminary phase and rank all 12 profiles. Only 4 profiles will be selected as candidates during the hiring decision.

The pre-screen process and the given information in the profiles vary treatments by treatment (see interventions).
Experimental Design Details
The preliminary phase is designed to generate actual profiles of candidates for use in the second phase of the hiring game. The benefit of using actual profiles is to introduce actual costs for discriminatory behaviour and therefore capture the actual level of employer discrimination (Hedegaard & Tyran, 2018). During this phase, we are going to ask participants to finish an individual experiment, including five individual tasks with 2 minutes each. The individual task is a 4-letter word anagram that participants need to correctly rearrange as many as possible sets of 4 letters to a meaningful word in 2 minutes. At the end of this phase, we will ask participants to finish an exit survey to capture their individual demographic differences. With five 4-letter anagram tasks, we were able to measure each individual's productivity and generate the inputs for the second phase of the hiring game.

The second phase is a hiring game, in which we will introduce four different treatments, a soft AA policy for an ethnic minority group, a soft AA policy for a randomly selected group, and baselines for both with and without information about ethnicity. In the second part of a hiring game, we will only recruit the majority as our participants. The majority are those who self-reported as White, currently live in the U.S., are born in the U.S., use English as their first native language. And all the participants for this experiment need to make two decisions: 1) hiring decision 2) estimation decision.

Prior to making a hiring decision, all profiles will go through a “pre-screen process" in which the computer will randomly select one of the five tasks completed by the individuals of profiles during the preliminary phase and rank all 12 profiles. Only 4 profiles will be selected as candidates during the hiring decision. During the hiring decision, participants will receive the profiles of four candidates, including scores of another drawn task (different from task used in “pre-screen process"), and age. Whether the ethnicity type of each candidate is included in the profiles, and the way for selecting 4 candidates' profiles are different treatments by treatments. We have 4 different treatments:

- Baseline (1): The profiles of the top 4 best performers will enter the candidate pool for all participants to choose from. During the selection,
the ethnic type will not be explicitly informed.
- Baseline – Type (2): The profiles of the top 4 best performers will enter the candidate pool for all participants to choose from. During the
selection, the ethnic type will be explicitly informed.
- Soft AA – Type (3): The profiles of the top 2 best-performing minorities and the top 2 best-performing non-selected remainders will enter the candidate pool. During the selection, the ethnic type will be explicitly informed.
- Soft AA – Lucky (4): Half of the profiles will be assigned as “lucky", and half will be assigned as “unlucky". The profile of the top 2 best performing individuals from the “lucky" group and the top 2 best-performing non-selected remainders will enter the candidate pool. During the selection, the ethnic type will not be explicitly informed.

Randomization Method
Randomisation was done by a computer through O-Tree
Randomization Unit
Individual participant
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
100 individual participants per treatment (500 in total, 400 for the main experiment)
Sample size: planned number of observations
12 profiles per session. 100 sessions per treatment. Total observations are 12*100*4 = 4800.
Sample size (or number of clusters) by treatment arms
100 sessions per treatment (total are 400 sessions)
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
IRB

Institutional Review Boards (IRBs)

IRB Name
IRB Approval Date
IRB Approval Number
Analysis Plan

Analysis Plan Documents

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials