Unveiling bias: the impact of blinded marking on university examen evaluation

Last registered on June 11, 2024

Pre-Trial

Trial Information

General Information

Title
Unveiling bias: the impact of blinded marking on university examen evaluation
RCT ID
AEARCTR-0013553
Initial registration date
May 21, 2024

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
May 29, 2024, 10:18 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
June 11, 2024, 4:15 PM EDT

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Region

Primary Investigator

Affiliation
Universidad de San Andrés

Other Primary Investigator(s)

PI Affiliation
Universidad de San Andrés
PI Affiliation
Universidad de San Andrés

Additional Trial Information

Status
In development
Start date
2024-08-05
End date
2024-12-23
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
This experiment at the University of San Andrés investigates the effects of blind versus non-blind grading methods on the persistence of grades across undergraduate courses, which are randomly distributed into control and treatment groups. In the control group, instructors can easily identify their students during grading, as students are required to write their full names on their exams. Conversely, in the treatment group, students use anonymized ID numbers, which obscures their identities from the graders. The study evaluates various outcomes, including the overall level of grades, fluctuations between midterm grades, alignment with historical GPA, and the incidence of repeating grades. Additionally, the research includes an analysis of how different grading methods influence these outcomes in relation to student characteristics such as gender, age, financial aid status, and special educational needs (SEN).
External Link(s)

Registration Citation

Citation
Llamas, Paola, Tomas Pacheco and Martín Rossi. 2024. "Unveiling bias: the impact of blinded marking on university examen evaluation." AEA RCT Registry. June 11. https://doi.org/10.1257/rct.13553-2.0
Experimental Details

Interventions

Intervention(s)
This experiment involves all undergraduate courses at the University of San Andrés. Courses are randomly assigned to two groups, treatment and control, using a computer. This experiment aims to analyze the persistence of grades within courses. That is, we want to analyze whether students' grades on the second midterm are positively related to the grades they received on the first midterm.
Control Group: Teachers will have the following information about their students: full name, email, major, and student photo. In these courses, when taking the first and second midterms, students will write their full names on their papers. Thus, teachers can fully identify their students while grading the exams. When grading the second midterm, they can also fully identify the students. That is, both assessments in the course are traditional (or non-blind).

Treatment Group: In courses assigned to the treatment group, teachers will have access to a complete list of their students from the beginning of the semester with the following information: name, email, major, and photo. In these courses, when taking the first midterm, students will write their student ID number (identifier) on their papers. Thus, teachers cannot fully identify their students while grading the exams. When grading the second midterm, students will write a different ID number on their papers. Thus, teachers cannot identify their students during both assessment instances.

This analysis will include an examination of differences in grades based on observable student characteristics such as gender, age, financial aid status (full scholarship), and special educational needs (SEN). We will show econometric results for all courses and without math courses.
Intervention Start Date
2024-09-30
Intervention End Date
2024-12-23

Primary Outcomes

Primary Outcomes (end points)
Grade; absolute difference between second midterm grade and first midterm; absolute difference between midterm grade and past GPA; Same grade.
Primary Outcomes (explanation)
1. Outcome 1: Grades
- Objective: Determine whether blind grading versus non-blind grading results in higher or lower average first midterm grades.

2. Outcome 2: Absolute Difference Between Second Midterm grade and First Midterm grade:
- Objective: Investigate whether the grading method (blind vs. non-blind) influences the discrepancy between midterm grades.

3. Outcome 3: Absolute Difference Between Midterm and GPA:
- Objective: Investigate whether the grading method (blind vs. non-blind) influences the discrepancy between midterm grades and students' historic GPA.

4. Outcome 4: Same grade.
- Objective: Investigate whether the grading method (blind vs. non-blind) influences the probability of a student receiving identical grades in both the first and second midterms.

5. Outcome 5: Change of grades.
- Objective: Investigate whether the grading method (blind vs. non-blind) influences the probability of a student having his/her grade changed in the first midterm.

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
We will take all undergraduate courses at Universidad de San Andrés and conduct a random assignment without stratification so that half of the classes must grade the blind exam and the other half do not. For this intervention, we will have N=x classes to draw, so x/2 will belong to the treated group and x/2 to the control group.
Experimental Design Details
Not available
Randomization Method
Randomization by a computer.
Randomization Unit
We will randomize classes without stratification. The analysis will be done at the student level.
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
We cluster at the class level (X classes).
Sample size: planned number of observations
X students
Sample size (or number of clusters) by treatment arms
X classes, X/2 treated, X/2 controls.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
IRB

Institutional Review Boards (IRBs)

IRB Name
IRB Approval Date
IRB Approval Number