Examining Bias in Teacher Evaluations of Students

Student grades in schools are supposed to be a reflection of the students’ real ability in the subject, and commonly decide which students go to which high school, higher education and job. As such, from a fairness perspective, it is important that student grades reflect the the student’s ability in the subject. To identify whether students are treated fairly, we recruit real teachers to evaluate real student assignments.

The assignments are randomly assigned student names, which are intended to signal the students’ ethnicity and gender. The identifica-
tion strategy is motivated by traditional CV-experiments, where job-applicants are randomly assigned names to study whether applicants with different names are treated unequally in job application processes. In these experiments, the ethnicity and gender of the job applicant
are traditionally signaled through a random assignment of names (See e.g., Midtbøen, 2015; Midtbøen, 2016; Ahmad, 2020). Our study also builds upon a small literature using comparable experiments to investigate discrimination in grading (see e.g., Hanna and Linden, 2012;
Sprietsma, 2013; Van Ewijk, 2011).

2023-11-23

2024-08-31

Our primary outcome is the assessment of students’ ability levels by teachers.

We collect this data through our questionnaire, in which teachers evaluate written assignments for an entire class using the standard Norwegian grading scale. The Norwegian grading scale spans from 1 to 6, with 6 representing the highest grade and 1 signifying a failed assignment. We collect student grades for both a Mathematics and a Norwegian language assignment.

The interventions will be randomized across teachers.

To study the effect of students’ ethnicity and gender on teacher evaluations, we randomly assign student names to actual assignments to investigate whether the name, as an indicator of ethnicity and gender, causally influences evaluations of the students’ abilities. The student names are easily visible to the participating teachers at the beginning of the student assignment, as well as where the grades are assigned.
Teachers are recruited through collaboration with multiple schools that assist in distributing the assignments and the questionnaire to them. All assessed assignments are completed by actual students, and each teacher is compensated economically for evaluating a complete class’ assignments. The assignment should prevent teachers from discerning the gender and ethnicity of the original student/author, and both the grading and post-grading questionnaires are distributed electronically via Qualtrics. In the questionnaire, both the order of the
assignments and the names on the assignments are randomized. The teacher will initially assign a grade, which can be adjusted in the final overview of grades for all the assignments. Therefore, we will have both the initial grade and the grades set after adjustments.

In the study, we establish a few control groups. The first control group involves teacher evaluations of assignments without any access to students’ names. This blind evaluation aims to assess students’ abilities impartially and is incorporated to enable us to estimate the
impact of gender and ethnicity in comparison to this untreated control group. Our second control group consists of assignments with native names, and it is employed as a control group in regression analyses where we estimate the influence of ethnicity. This allows us to assess the impact of ethnicity in comparison to both a blind control group and a control group with Norwegian names. In our examination of gender effects, we utilize male and female names for both native and immigrant students. Female names serve as the control group when estimating the impact of having a male name on the assignment. We therefore evaluate the gender effect for all assignments collectively, as well as separately for assignments with Norwegian names and those with non-native names. This approach allows us to assess whether there is variation in gender bias between native and non-native names.

Randomization done in office by a computer.

30 student assignments

No

Our goal is to recruit approximately 150 teachers for the project, with an anticipated participation
of over 100 teachers who will each grade approximately 30 assignments. This amounts to a
total of roughly 3,000 graded assignments clustered at the teacher level.

Our goal is to recruit approximately 150 teachers for the project, with an anticipated participation of over 100 teachers who will each grade approximately 30 assignments. This amounts to a total of roughly 3,000 graded assignments.

Each teacher will grade an entire class of student assignments. This amounts to a sample size equal to the number of teachers per name that is randomly assigned. Our goal is to recruit approximately 150 teachers for the project, with an anticipated participation of over 100 teachers who will each grade approximately 30 assignments. This amounts to a total of roughly 3,000 graded assignments.

With 3,000 observations, we have a statistical power of 0.8 to detect an effect size of .004 (Cohen’s f2) at a significance level (α) of 0.05.

Pre-Analysis Plan

Updated Pre-Analysis Plan

Note detailing the date of updates

We investigate the impact of ethnicity and gender on teacher assessments through a randomized controlled experiment involving 203 teachers grading a single class's assignments. Within teachers we randomly assigned names, signaling gender and ethnicity, with a control group grading anonymized assignments. We find no statistically significant evidence of grading bias based on student name, ethnicity or gender, unlike previous studies. This holds even among teachers with stronger implicit biases against foreign or male names. However, we observe significant variation in grades assigned by teachers for identical assignments. Finally, through counterfactual policy simulations, we demonstrate an economically relevant trade-off between precision and accuracy for grading policies in schools.

Alne, Ragnar H., Herstad, Eyo I. and Reiling, Rune B., Fairness in Grading: Randomizing Ethnicity and Gender in Teacher Assessments (October 28, 2024). Available at SSRN: https://ssrn.com/abstract=5002237 or http://dx.doi.org/10.2139/ssrn.5002237

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5002237