Disentangling Sources of Bias: Evidence from Advanced Placement Course Recommendations

The plan for this study is to examine whether school counselors are more or less likely to recommend student (as represented by their academic transcripts) for advanced placement (AP) calculus courses based on the race and gender suggested by the student names, and how prepared the school counselor believes the student would be for the AP calculus course. If there is race or gender bias in the likelihood of recommendation or the preparedness rating, is it possible to identify the source of the bias (statistical discrimination, implicit bias, or taste-based discrimination) by implementing experimental interventions and survey based measurements of those sources?

School counselors will be sent a survey to judge preparedness and likelihood of recommendation for advanced math courses. These surveys will have transcripts of students that excelled both academically and behaviorally, as well as borderline cases in one or both category. Names will be assigned randomly to these transcripts. The names will connote both a race/ethnicity and a socioeconomic status (where applicable). The aim of the survey is to understand if school counselors exhibit bias based on race/ethnicity, gender, or perceived socioeconomic status. A blinded transcript with no identifying information will be used as a control and 2 transcripts with racially ambiguous names will be used as baselines that all participants will receive. Counselors will be asked to judge six total profiles (2 baseline + 4 experimental profiles).

To identify the sources of bias we included three additional experimental conditions. First, we ask every participant to take an Implicit Associations Test (IAT) at the end of the experiment. This test provides a measure of a participants’ implicit bias. Second, we randomly included additional academic information in the form of a high math PSAT score for a subsample of survey participants. If statistical discrimination is a source of bias, the introduction of additional positive academic information should decrease any evaluation gaps by race or gender on average. Finally, we included a series of questions vetted in the psychology literature to measure taste-based discrimination.

2023-09-01

2024-06-30

1) recommendation for AP Calculus (yes or no)
2) preparedness for AP Calculus (scale of 1-10)

To answer the proposed research questions, we conduct a randomized experiment that is a modification of a correspondence audit study. We ask study participants to evaluate a set of 6 profiles which are given at random either a Black, White, Hispanic, or Asian sounding name, or no name (blind review). There are also typical male or female sounding names within each racial/ethnic category. For the Black and White-sounding names, we also choose names indicative of high, medium, or low socioeconomic status as demonstrated by sociological research on names.

To answer the proposed research questions, we conduct a randomized experiment that is a modification of a correspondence audit study. We ask study participants to evaluate a set of 6 profiles which are given at random either a Black, White, Hispanic, or Asian sounding name, or no name (blind review). There are also typical male or female sounding names within each racial/ethnic category. For the Black and White-sounding names, we also choose names indicative of high, medium, or low socioeconomic status as demonstrated by sociological research on names.

No individual study participant will be given an identical profile with only a changed name; rather the study relies on aggregate data from all participants. Since the only differences in the profiles will be the gender and racial origin of the names on top, we would expect no significant difference in the average likelihood of study participants recommending a particular student profile for advanced coursework. Significant differences in the likelihood of being recommended for AP Calculus across ethnic and gender name groups will be taken as evidence of bias (positive or negative). The blind review is included as an experimental control.

For a randomized subset of participants, we will implement a treatment designed to test for statistical discrimination. Specifically, we will add the information that the student received a high or average PSAT math score to the academic profile information. If statistical discrimination is a source of bias, the introduction of additional positive academic information should decrease any evaluation gaps by race or gender on average.

We will also collect demographic data on study participants that included race, gender, and years of experience as a school counselor and/or teacher. And we will match (and then de-identify) the school counselor with U.S. Department of Education data on their school of employment. We will use this information to examine whether particular counselor characteristics such as experience, training, or gender and school context characteristics such as #of AP courses offered or school demographic composition are associated with higher or lower estimates of bias.

Finally, we administer an implicit association test (IAT) designed to measure implicit or unconscious bias, and a survey instrument designed to measure taste-based discrimination.

There are two levels of randomization. Assignment to blinded transcripts without PSAT scores (control), blinded transcripts with PSAT scores (Treatment A), race/gender transcripts without PSAT scores (Treatment B) or race/gender transcripts with PSAT scores (Treatment C) will be randomized. Once assigned to control, treatment A, treatment B, or treatment C, the type of transcript presented for each of the 4 treatment transcripts (strong or borderline) will be randomized and a for those in the race/gender treatments, a randomized race/gender signifying name will be paired with the randomized transcript.

The randomization will be conducted by Qualtrics survey software .

Individual. There are two levels of randomization. Assignment to blinded transcripts without PSAT scores (control), blinded transcripts with PSAT scores (Treatment A), race/gender transcripts without PSAT scores (Treatment B) or race/gender transcripts with PSAT scores (Treatment C) will be randomized. Once assigned to control, treatment A, treatment B, or treatment C, the type of transcript presented for each of the 4 treatment transcripts (strong or borderline) will be randomized and a for those in the race/gender treatments, a randomized race/gender signifying name will be paired with the randomized transcript.

Yes

1300 School Counselors

5232 transcript ratings over 1308 School Counselors

Blinded transcripts without PSAT scores - 65 counselors (260 transcripts)
Blinded transcripts with PSAT scores - 65 counselors (260 transcripts)
Race/gender transcripts without PSAT scores - 589 counselors (2356 transcripts)
Race/gender transcripts with PSAT scores - 589 counselors (2356 transcripts)

Using a participant-level randomized cluster, and treatment at level 2, we conducted a power calculation to determine the number of clusters needed to have a sufficiently powered study. We used a standard alpha of .05. Estimated effect size was drawn directly from the pilot study (Francis, de Oliveira, and Dimmitt, 2019). Due to the overall goals of the future study, we used the effect size of Black Female students in assignment to AP coursework, which had an effect size of 0.2. The calculated variance in effect size was 0.32. The r-squared was also pulled directly from the pilot study at 0.249. The sample size is determined by the number of tests within the cluster, each cluster has four experimental evaluations (not counting the two baseline evaluations). Using Optimal Design software, this presented us with a needed sample size of 218 evaluations per hypothesis. With 2 racial groups, 2 genders, 3 SES levels, and 2 statistical discrimination groupings, we have 24 total hypotheses (2x2x3x2) for our main analysis. The required number of evaluations would be 5,232. Each counselor will conduct four experimental evaluations for a total of 1,308 counselors required.