Feedback, Motivation, and Performance

Last registered on September 26, 2023

Pre-Trial

Trial Information

General Information

Title
Feedback, Motivation, and Performance
RCT ID
AEARCTR-0010898
Initial registration date
August 25, 2023

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
September 04, 2023, 6:03 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
September 26, 2023, 11:20 AM EDT

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Region

Primary Investigator

Affiliation
University of Florida

Other Primary Investigator(s)

PI Affiliation
University of Florida
PI Affiliation
University of Florida
PI Affiliation
University of Florida

Additional Trial Information

Status
On going
Start date
2023-08-25
End date
2023-12-20
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
Individuals often make educational and professional decisions conditional on feedback of their performance. In educational settings, students receive grades as the primary source of feedback. These grades may impact how students choose to allocate their efforts, as well as influence future educational decisions (major, course enrollment, etc.). The most challenging issue in this line of research is the lack of a exogenous variation in feedback. We aim to overcome this selection issue by adopting an experimental design in which we randomly assign graders of varying innate harshness (despite consistent guidelines) to blindly grade short essays in two large introductory economics courses. With this, we are able to analyze the causal effect of a grade’s harshness/lenience on students’ subsequent decisions and outcomes. We also measure students’ grade expectations (confidence) and track grade challenges, enabling us to characterize a potential casual channel (confidence) and identify heterogeneities in the resistance to feedback.
External Link(s)

Registration Citation

Citation
Knight, Thomas et al. 2023. "Feedback, Motivation, and Performance." AEA RCT Registry. September 26. https://doi.org/10.1257/rct.10898-1.1
Experimental Details

Interventions

Intervention(s)
Intervention Start Date
2023-09-16
Intervention End Date
2023-11-20

Primary Outcomes

Primary Outcomes (end points)
1- Course Performance: Exam grades. Within several days after each short writing assignment’s grades are posted, students will be taking exams consisting of 40 multiple choice questions.
2- Subsequent Educational Decisions: Major choice, course enrollment in future semesters. We will request from the University Registrar students’ declared major and course enrollment in the semester following the experiment.
3- Grade disputes: Students are allowed to reach out to the instructor if they would like to ask for a regrading of any part of their assignments. We collect information on what part of their assignment they challenge their grades and whether it was a success as well as the text of the email sent to the instructor to obtain sentiment scores in these texts.
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
1- Course Participation and Engagement: Learning Management System (LMS) activity and resource utilization. The learning management system collects hourly-aggregated page and file view statistics for all students (with time stamps for first and last view within the hour). We will use these statistics to construct several measures of course engagement. Active days are the number of days in a period a student viewed any page or resource on Canvas. Using time stamps, we will also measure procrastination (how late after an assignment is first available that a student first works on it).
2- Expected Performance: After completing each short writing assignment and exam, students are asked to provide an incentivized estimate of their expected grade.
3- Demand for Feedback: time to view posted grade, answer key. Using course activity statistics provided by the LMS, we will measure the demand for simple feedback continuously as the time to view a grade after it has been posted. For exams, grades will be posted concurrently with an answer key. Viewing this resource will be interpreted as a demand for detailed feedback (binary).
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
This is a field experiment conducted in two large principles of economics courses at a large, comprehensive university. Students will be asked to complete 3 short writing assignments (SWAs), which are short essays answering some content-relevant prompt. All prompts have an objectively correct answer in the rubric provided to all graders, but partial credit can be awarded at the discretion of the grader. For each assignment, students are also asked to provide a grade expectation, the honesty of which is incentivized by providing extra credit for accuracy.
Once the assignment due date has passed, students’ assignments are randomly assigned to a grader. Graders are responsible for only grading and do not interact with students in any way. Grading is double-blind such that neither student nor grader is aware of the identity of the other during grading period until grades are posted. Graders are provided a standard rubric of ten elements, all valued at ten points each. The first seven are objective and related to the SWA prompt. The final three elements are concerned with grammar, typos, and writing quality. All graders are given guidelines on the grading of these elements at the start of the semester. However, all entered grades are at the sole discretion of the grader and are never altered by anyone other than the grader or the instructor. Once all assignments have been graded, all detailed rubrics are downloaded from the LMS before being posted. Students are immediately notified of posted grades by the LMS. At the same time, the rubric is available to students so that they can review the points they lost, if any. Once posted, the assigned grade is no longer double-blind as the student is now able to view the grader’s name. Grader's names seen by students are randomly assigned to a generic female and male sounding names (Amanda, Emma, Eric, James, Jessica, John, Katherine, and Michael).
Students are permitted to contest a grade only by contacting the course instructor. Graders are never permitted to interact with students and are instructed to forward any student communication to the study coordinator. The instructor has the sole discretion to alter any submitted grade. Instructors maintain a record of all grade contests, and any alterations are flagged when rubrics are downloaded at the end of the semester.
Several days after each short writing assignment, students complete a 40-question multiple choice exam covering the same material as the preceding SWA. The final question of each exam asks students to report the number of questions they expect to get correct. Similar to the SWAs, honesty is incentivized by providing extra credit for accuracy.
After the assignment grades have been posted for the first and second assignments, students will be asked to provide a rating of their perceived fairness of the each grade they were assigned (1 to 5 where 1 is “Very Unfair” and 5 is “Very Fair”). This response is not incentivized. On the final exam, students will be asked to provide a rating of their perceived fairness of the overall grading in the course (1 to 5 where 1 is “Very Unfair” and 5 is “Very Fair”).
At the end of the semester, each SWA will be regraded blindly by two additional randomly-assigned graders.
Experimental Design Details
Randomization Method
We randomize treatment assignment at the individual level. Randomization by the computer.
Randomization Unit
Individual participant.
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
3 Short Writing Assignments for each participant, for approximately 7122 observations (expected to change based on drop/adds)
Sample size: planned number of observations
3 Short Writing Assignments for each participant, for approximately 7122 observations (expected to change based on drop/adds)
Sample size (or number of clusters) by treatment arms
3 Short Writing Assignments for each participant, for approximately 7122 observations (expected to change based on drop/adds).
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
IRB

Institutional Review Boards (IRBs)

IRB Name
University of Florida
IRB Approval Date
2022-12-01
IRB Approval Number
IRB202201907

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials