Disentangling Sources of Bias: Evidence from Advanced Placement Course Recommendations

Last registered on November 21, 2021

Pre-Trial

Trial Information

General Information

Title
Disentangling Sources of Bias: Evidence from Advanced Placement Course Recommendations
RCT ID
AEARCTR-0007927
Initial registration date
November 16, 2021
Last updated
November 21, 2021, 11:16 AM EST

Locations

There are documents in this trial unavailable to the public. Use the button below to request access to this information.

Request Information

Primary Investigator

Affiliation
University of Massachusetts Boston

Other Primary Investigator(s)

PI Affiliation
University of Massachusetts Amherst
PI Affiliation
University of Massachusetts Amherst

Additional Trial Information

Status
In development
Start date
2022-01-01
End date
2022-04-30
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
In this project, we seek to understand minority and female underrepresentation in advanced STEM courses in high school by investigating whether school counselors exhibit racial or gender bias during the course assignment process. We extend the analysis by attempting to disentangle whether any observed bias can be attributed to taste-based discrimination, statistical discrimination, or implicit bias. Using an adapted audit study, we asked a nationally-recruited sample of school counselors to evaluate student transcripts that were identical except for the names on the transcripts, which were varied randomly to suggestively represent a chosen race and gender combination. In order to identify the sources of bias we included three additional experimental conditions. Understanding the underlying sources of racial and gender bias can help stakeholders and policymakers design better solutions to address the bias.
External Link(s)

Registration Citation

Citation
de Oliveira, Angela, Carey Dimmitt and Dania Francis. 2021. "Disentangling Sources of Bias: Evidence from Advanced Placement Course Recommendations." AEA RCT Registry. November 21. https://doi.org/10.1257/rct.7927-1.0
Experimental Details

Interventions

Intervention(s)
The plan for this study is to examine whether school counselors are more or less likely to recommend student (as represented by their academic transcripts) for advanced placement (AP) calculus courses based on the race and gender suggested by the student names, and how prepared the school counselor believes the student would be for the AP calculus course. If there is race or gender bias in the likelihood of recommendation or the preparedness rating, is it possible to identify the source of the bias (statistical discrimination, implicit bias, or taste-based discrimination) by implementing experimental interventions and survey based measurements of those sources?
Intervention Start Date
2022-01-01
Intervention End Date
2022-04-30

Primary Outcomes

Primary Outcomes (end points)
1) recommendation for AP Calculus (yes or no)
2) preparedness for AP Calculus (scale of 1-10)
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)

Secondary Outcomes (explanation)

Experimental Design

Experimental Design
To answer the proposed research questions, we conduct a randomized experiment that is a modification of a correspondence audit study. We ask study participants to evaluate a set of 6 profiles which are given at random either a Black, White, Hispanic, or Asian sounding name, or no name (blind review). There are also typical male or female sounding names within each racial/ethnic category. For the Black and White-sounding names, we also choose names indicative of high, medium, or low socioeconomic status as demonstrated by sociological research on names.

Experimental Design Details
Not available
Randomization Method
The randomization will be conducted by Qualtrics survey software .
Randomization Unit
Individual. There are two levels of randomization. Assignment to blinded transcripts without PSAT scores (control), blinded transcripts with PSAT scores (Treatment A), race/gender transcripts without PSAT scores (Treatment B) or race/gender transcripts with PSAT scores (Treatment C) will be randomized. Once assigned to control, treatment A, treatment B, or treatment C, the type of transcript presented for each of the 4 treatment transcripts (strong or borderline) will be randomized and a for those in the race/gender treatments, a randomized race/gender signifying name will be paired with the randomized transcript.
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
240 School Counselors
Sample size: planned number of observations
960 transcript ratings over 240 School Counselors
Sample size (or number of clusters) by treatment arms
Blinded transcripts without PSAT scores - 60 school counselors
Blinded transcripts with PSAT scores - 60 school counselors
Race/gender transcripts without PSAT scores - 60 school counselors
Race/gender transcripts with PSAT scores - 60 school counselors
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Preparedness Rating (value 1-10): Minimal Detectable Effect Size: 0.8 points (from pilot study'0 Percentage: 80% Alpha: 0.05 two-tailed comparison In order to capture the 3 levels of SES, the sample size would need to be 876 transcript ratings. However, with 4 treatments per survey (and assuming 876 as our goal), that would require distribution of 219 surveys. We are aiming for 240 completed surveys.
Supporting Documents and Materials

There are documents in this trial unavailable to the public. Use the button below to request access to this information.

Request Information
IRB

Institutional Review Boards (IRBs)

IRB Name
University of Massachusetts Amherst Human Research Protection Office
IRB Approval Date
2019-04-03
IRB Approval Number
937