Disentangling Sources of Bias: Evidence from Advanced Placement Course Recommendations

The plan for this study is to examine whether school counselors are more or less likely to recommend student (as represented by their academic transcripts) for advanced placement (AP) calculus courses based on the race and gender suggested by the student names, and how prepared the school counselor believes the student would be for the AP calculus course. If there is race or gender bias in the likelihood of recommendation or the preparedness rating, is it possible to identify the source of the bias (statistical discrimination, implicit bias, or taste-based discrimination) by implementing experimental interventions and survey based measurements of those sources?

2023-09-01

2024-06-30

1) recommendation for AP Calculus (yes or no)
2) preparedness for AP Calculus (scale of 1-10)

To answer the proposed research questions, we conduct a randomized experiment that is a modification of a correspondence audit study. We ask study participants to evaluate a set of 6 profiles which are given at random either a Black, White, Hispanic, or Asian sounding name, or no name (blind review). There are also typical male or female sounding names within each racial/ethnic category. For the Black and White-sounding names, we also choose names indicative of high, medium, or low socioeconomic status as demonstrated by sociological research on names.

Not available

The randomization will be conducted by Qualtrics survey software .

Individual. There are two levels of randomization. Assignment to blinded transcripts without PSAT scores (control), blinded transcripts with PSAT scores (Treatment A), race/gender transcripts without PSAT scores (Treatment B) or race/gender transcripts with PSAT scores (Treatment C) will be randomized. Once assigned to control, treatment A, treatment B, or treatment C, the type of transcript presented for each of the 4 treatment transcripts (strong or borderline) will be randomized and a for those in the race/gender treatments, a randomized race/gender signifying name will be paired with the randomized transcript.

Yes

1300 School Counselors

5232 transcript ratings over 1308 School Counselors

Blinded transcripts without PSAT scores - 65 counselors (260 transcripts)
Blinded transcripts with PSAT scores - 65 counselors (260 transcripts)
Race/gender transcripts without PSAT scores - 589 counselors (2356 transcripts)
Race/gender transcripts with PSAT scores - 589 counselors (2356 transcripts)

Using a participant-level randomized cluster, and treatment at level 2, we conducted a power calculation to determine the number of clusters needed to have a sufficiently powered study. We used a standard alpha of .05. Estimated effect size was drawn directly from the pilot study (Francis, de Oliveira, and Dimmitt, 2019). Due to the overall goals of the future study, we used the effect size of Black Female students in assignment to AP coursework, which had an effect size of 0.2. The calculated variance in effect size was 0.32. The r-squared was also pulled directly from the pilot study at 0.249. The sample size is determined by the number of tests within the cluster, each cluster has four experimental evaluations (not counting the two baseline evaluations). Using Optimal Design software, this presented us with a needed sample size of 218 evaluations per hypothesis. With 2 racial groups, 2 genders, 3 SES levels, and 2 statistical discrimination groupings, we have 24 total hypotheses (2x2x3x2) for our main analysis. The required number of evaluations would be 5,232. Each counselor will conduct four experimental evaluations for a total of 1,308 counselors required.