Weight bias in SET scores

Last registered on October 04, 2023

Pre-Trial

Trial Information

General Information

Title
Weight bias in SET scores
RCT ID
AEARCTR-0012189
Initial registration date
September 26, 2023

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
October 04, 2023, 2:02 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Region

Primary Investigator

Affiliation
Management Development Institute Gurgaon

Other Primary Investigator(s)

PI Affiliation
Ahmedabad University

Additional Trial Information

Status
In development
Start date
2023-09-27
End date
2024-09-27
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
We examine whether students are weight-biased against professors and whether a professor's gender mitigate (or amplify) that bias in SET (Student Evaluation of Teaching) scores. We also examine the role of student gender in our expected weight bias.
External Link(s)

Registration Citation

Citation
Arora, Puneet and Moumita Roy. 2023. "Weight bias in SET scores." AEA RCT Registry. October 04. https://doi.org/10.1257/rct.12189-1.0
Experimental Details

Interventions

Intervention(s)
We conduct our experiment at the beginning of the bi-semester 2023-2024 (end of September) with students enrolled in the 7 sections of Principles of Microeconomics course. The experiment will be conducted in the first class of the course where students will be taught the same topic by normal weight and overweight/obese looking instructors. We conduct the intervention for both male and female instructors to see whether the effect differs by gender.

Intervention Start Date
2023-09-27
Intervention End Date
2023-10-04

Primary Outcomes

Primary Outcomes (end points)
1. SET scores as reflected in six variables: Quality of instructional material, Teaching Effectiveness, Preparation and Organization of class, Overall an interesting lecture, Overall evaluation of instructor.
2. Average SET score over all 13 questions asked in the evaluation form
3. Students' test scores on the quiz conducted at the end of the experimental lecture

Primary Outcomes (explanation)
We will ask seven questions that relate to teaching effectiveness, we will take their average to create the Teaching Effectiveness variable

We will ask two questions that relate to Preparation and Organization of class, , we will take their average to create the Preparation and Organization of class variable

We will ask a total of 13 questions on the SET form, and we will average those to create the Average SET score

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
The experiment will be conducted in the first lecture of the Principles of Microeconomics course, where students will be randomly assigned to four treatment groups, with each treatment sitting in separate classrooms. There are close to 500 enrolled students in the course. We will conduct the experiment in multiple classrooms (and multiple slots if required) depending on student availability.

Students will pre-registered their preferred/available slot, where we collect information on their gender too. We will randomize them individually into the four treatments, stratified by gender using Stata's randomization program.

During the experiment, they will attend the lecture, followed by a short 8 questions multiple choice question quiz, followed by the SET form, demographics form and their beliefs to understand the role of social norms about female/male as professors and an individual's weight (fatness) as an impediment to success.

The experiment will be conducted in-person, with the lecture played on the projector, and quiz/SET/demographics/belief form created over google form, sent to each student over email to be filled in the class at the end of the lecture.
Experimental Design Details
We propose to conduct our experiment at the beginning of the bi-semester 2022-2023 with students enrolled in the 7 sections of Principles of Microeconomics course. The experiment will be conducted in the first class of the course. We will randomly divide all students into four equal groups, and each of these groups will sit together in their assigned classroom only for that one class.

This first class, which will incorporate our intervention, will not be conducted by their regular instructor. The session will be for 1 hour and 15 minutes, where students will attend a recorded video lecture which will introduce them to the early concepts in economics for 20 minutes, students will then take an 8 question multiple choice question quiz for 10 minutes, and will then fill the teaching evaluation (13 questions mapping into 6 broader teaching dimensions discussed above) and demographics form (information on age, gender, programme of study, major, parental education, household income-bracket, etc.), ending with the beliefs survey about their perception of social norms about male/female's appropriateness as professors, and their belief about role of an individual's weight as an impediment in their success. The belief survey will specifically ask students about their expected grade in the course, about how they think male students generally perceive the appropriateness of a male and of a female as professors. Similarly, how female students generally perceive the appropriateness of a male and of a female as professors. We will also ask them three questions about how they perceive fat people (asking them to rate their belief about: whether "fat people cannot be successful as others", "fat people cause their own problems by not exercising", "fat people have different personalities than other people"). These questions are asked in order to see whether students' perception of social norms with respect to the appropriateness of the gender of a professor or his/her weight affects how the student rates that professor.

Quiz, teaching evaluation and demographics questions will be asked anonymously over Google Form after the 20 minutes audio lecture , without asking for subject's name or university id. Students will be articulately informed that as instructors, we are interested in their average understanding of the topic and their average feedback of the instructor, and therefore, all the information being collected is anonymous.

The four groups created in the experiment will include T1: Male-Normal weight type, where a male instructor with normal looking weight will teach. The other treatment groups include T2: Female-Normal weight type, where a female instructor with normal looking weight will teach; T3: Male-Overweight type, where a male instructor with observably overweight weight body type will teach; and T4: Female-Overweight type, where a female instructor with observably overweight weight body type will teach.

In each treatment, the instructor will start by introducing himself/herself for two minutes, before starting with the lecture. The information introducing the instructor will include instructor's PhD granting institution (a US university name used to reflect the typical internationally education faculty at this institution), teaching experience, research interests, and professional service engagement. We will use hypothetical names for the instructors (Amit Agarwal for male and Sunita Sharma for female), and the information curated will be such that it reflects the profile of a typical young faculty teaching such courses at the university. The experiment, however, will be conducted like a natural field experiment and students will not know the hypothetical nature of such information. This is to prevent any bias that provision of such information may cause with their regular instructor for the course. Students will, however, be sent a debriefing email at the end of the experiment informing them of the intent of the study, and to request their consent allowing us to use their data in the study.

In order to estimate the causal effect of gender*weight, we record the lecture by a course instructor, and modulate the voice using a software to create a male and a female version of the same lecture. We do not use the lecture recorded in the original voice. This design aspect ensures that teaching style and effectiveness is same in all four groups.

This implies that the differences between T1 and T3 groups' SET scores are attributed to weight differential among male instructors of otherwise identical teaching capabilities. Similarly, differences between T2 and T4 groups' SET scores are attributed to weight differential among female instructors of otherwise identical teaching capabilities.

Since the teaching is identical across the four groups. A difference between T1 and T2 will be indicative of gender-bias in SET scores by students when the instructors are normal-weight type. A difference between T3 and T4 will be indicative of gender-bias in SET scores by students when the instructors are overweight type.

Randomization Method
Randomization conducted using Stata
Randomization Unit
Individual level randomization, with stratification by gender
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
450+
Sample size: planned number of observations
450+
Sample size (or number of clusters) by treatment arms
110+ students in each of the four treatment arms
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
With our expected sample size, we should be able to detect a 0.2 sd weight-bias effect with 80% statistical power. Prior research (Boring and Philippe, 2021) shows that the gender bias in SET scores are expected to be larger than 0.14sd. Our own research (working paper Arora and Roy, 2023) also shows such expected effect to be greater than 0.2sd. We expect to find similar levels of weight-bias effect size in our study.
IRB

Institutional Review Boards (IRBs)

IRB Name
Ahmedabad University IRB
IRB Approval Date
2023-09-25
IRB Approval Number
N/A
Analysis Plan

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials