Role of Quality Signals in Student Evaluations of Teaching

Last registered on July 25, 2025
View Trial History

Pre-Trial

Trial Information

General Information

Title
Role of Quality Signals in Student Evaluations of Teaching
RCT ID
AEARCTR-0016414
Initial registration date
July 18, 2025

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
July 25, 2025, 11:28 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Country
India
Region

Primary Investigator

Name
Puneet Arora
Affiliation
Management Development Institute Gurgaon
Contact Primary Investigator

Other Primary Investigator(s)

Additional Trial Information

Status
In development
Start date
2025-07-21
End date
2025-12-31
Keywords
Behavior, Education, Gender
Additional Keywords
JEL code(s)
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
Student Evaluations of Teaching (SET) are one of the most widely used tools for assessing faculty performance in higher education. Despite their popularity, substantial research has documented concerns around their fairness, particularly highlighting that SET scores may reflect student biases rather than objective teaching quality. Gender-based disparities are of particular concern: female faculty members often receive lower ratings than their male counterparts even when teaching performance is equivalent. To counteract such bias, a commonly recommended strategy is for instructors, especially from underrepresented groups, to signal their professional quality through their academic qualifications, research accomplishments, and work experience. This approach is believed to offset the effects of stereotypes and provide students with a more complete view of the faculty member's merit.

However, one potential concern is whether such quality signaling has differential effects depending on the gender of the faculty member and the student. While signaling may help faculty from disadvantaged groups like women, the same strategy may backfire when used by faculty from traditionally advantaged groups such as men. In particular, there may be a perception that quality signals from male faculty constitute flaunting, which could adversely affect SET scores. Moreover, student gender may further moderate these effects, e.g. female students may be more sensitive to such dynamics than male students. This study conducts an experimental study to examine this interaction between faculty and student gender, and the impact of quality signaling, on SET scores in a higher education setting. I also examine the underlying mechanisms driving such interactions, if any.
External Link(s)

Registration Citation

Citation
Arora, Puneet. 2025. "Role of Quality Signals in Student Evaluations of Teaching." AEA RCT Registry. July 25. https://doi.org/10.1257/rct.16414-1.0
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
Information about faculty credentials/quality shared with some, while not shared with others.
Intervention (Hidden)
The study will be conducted at a business school, and will involve MBA students across multiple cohorts and course sections. Data collection will begin after students have attended at least four lectures of a selected course, ensuring they have adequate exposure to the instructor. Students will be invited to participate during their break time (eg, lunch break or at the end of all the lectures), with course instructor not present while students rate the faculty. Student participation will be sought only for those courses, and in those cohort-sections, where the course instructor has granted prior approval to the researcher (their assent will be documented over email), and students from any given cohort-section will participate in the study only once to avoid contamination. In other words, one student will not rate more than one instructor.

Students in each participating section will be randomly assigned to one of two experimental conditions using Qualtrics’ built-in randomization algorithm:
• Information Treatment: Students in this group will first view a slide presenting detailed information about the faculty member’s academic and professional credentials. This may include the faculty's PhD institution, research publications, industry consultancy experience, teaching awards, and other notable achievements.
• No-Information Treatment Group: Students in this group will proceed directly to the evaluation without receiving any additional information about the faculty member.

Following the treatment exposure, all students will complete the Student Evaluation of Teaching (SET) survey. The survey will be administered anonymously via Qualtrics, with no collection of personally identifying information (such as name, email ID, or roll number) to ensure truthful and unbiased responses.
Intervention Start Date
2025-07-22
Intervention End Date
2025-12-31

Primary Outcomes

Primary Outcomes (end points)
SETs
Primary Outcomes (explanation)
This study seeks to understand how students interpret and react to faculty quality signals in the context of SET, especially when these signals interact with gender-based perceptions. The study focuses on three core questions:
1. Do quality signals influence student evaluations (SET), and does this depend on the gender of the faculty member?
2. Are faculty members evaluated differently by students of the same gender (in-group bias) versus those of the opposite gender (out-group bias)?

Secondary Outcomes

Secondary Outcomes (end points)
Underlying mechanisms
Secondary Outcomes (explanation)
What are the underlying mechanisms driving these effects? (a) Perception of a particular gender faculty belonging to an advantaged or disadvantaged group; (b) Perception of one’s own gender belonging to an advantaged or disadvantaged group; (c) Appreciation of effort required to become an accomplished faculty; (d) Gap in expected and perceived actual teaching quality; (e) Perception of quality signaling as flaunting; and (f) Role of stereotyped beliefs.

Experimental Design

Experimental Design
The SETs will be conducted in the middle of the course (economics and other management courses, done only once with a cohort-section), and when students click on the survey, they'll be randomly assigned to one of two experimental conditions using Qualtrics’ built-in randomization algorithm:

• Information Treatment: Students in this group will first view a slide presenting detailed information about the faculty member’s academic and professional credentials. This may include the faculty's PhD institution, research publications, industry consultancy experience, teaching awards, and other notable achievements.

• No-Information Treatment Group: Students in this group will proceed directly to the evaluation without receiving any additional information about the faculty member.
Experimental Design Details
Randomization Method
Qualtrics in built randomization algorithm
Randomization Unit
Student level
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
500+
Sample size: planned number of observations
500+
Sample size (or number of clusters) by treatment arms
250+ each
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Supporting Documents and Materials

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
IRB

Institutional Review Boards (IRBs)

IRB Name
MDI IRB
IRB Approval Date
2025-07-18
IRB Approval Number
MDI IRB #2025-20
Analysis Plan

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials