Feedback, Motivation, and Performance

2023-09-16

2023-11-20

1- Course Performance: Exam grades. Within several days after each short writing assignment’s grades are posted, students will be taking exams consisting of 40 multiple choice questions.
2- Subsequent Educational Decisions: Major choice, course enrollment in future semesters. We will request from the University Registrar students’ declared major and course enrollment in the semester following the experiment.
3- Grade disputes: Students are allowed to reach out to the instructor if they would like to ask for a regrading of any part of their assignments. We collect information on what part of their assignment they challenge their grades and whether it was a success as well as the text of the email sent to the instructor to obtain sentiment scores in these texts.

1- Course Participation and Engagement: Learning Management System (LMS) activity and resource utilization. The learning management system collects hourly-aggregated page and file view statistics for all students (with time stamps for first and last view within the hour). We will use these statistics to construct several measures of course engagement. Active days are the number of days in a period a student viewed any page or resource on Canvas. Using time stamps, we will also measure procrastination (how late after an assignment is first available that a student first works on it).
2- Expected Performance: After completing each short writing assignment and exam, students are asked to provide an incentivized estimate of their expected grade.
3- Demand for Feedback: time to view posted grade, answer key. Using course activity statistics provided by the LMS, we will measure the demand for simple feedback continuously as the time to view a grade after it has been posted. For exams, grades will be posted concurrently with an answer key. Viewing this resource will be interpreted as a demand for detailed feedback (binary).

This is a field experiment conducted in two large principles of economics courses at a large, comprehensive university. Students will be asked to complete 3 short writing assignments (SWAs), which are short essays answering some content-relevant prompt. All prompts have an objectively correct answer in the rubric provided to all graders, but partial credit can be awarded at the discretion of the grader. For each assignment, students are also asked to provide a grade expectation, the honesty of which is incentivized by providing extra credit for accuracy.
Once the assignment due date has passed, students’ assignments are randomly assigned to a grader. Graders are responsible for only grading and do not interact with students in any way. Grading is double-blind such that neither student nor grader is aware of the identity of the other during grading period until grades are posted. Graders are provided a standard rubric of ten elements, all valued at ten points each. The first seven are objective and related to the SWA prompt. The final three elements are concerned with grammar, typos, and writing quality. All graders are given guidelines on the grading of these elements at the start of the semester. However, all entered grades are at the sole discretion of the grader and are never altered by anyone other than the grader or the instructor. Once all assignments have been graded, all detailed rubrics are downloaded from the LMS before being posted. Students are immediately notified of posted grades by the LMS. At the same time, the rubric is available to students so that they can review the points they lost, if any. Once posted, the assigned grade is no longer double-blind as the student is now able to view the grader’s name. Grader's names seen by students are randomly assigned to a generic female and male sounding names (Amanda, Emma, Eric, James, Jessica, John, Katherine, and Michael).
Students are permitted to contest a grade only by contacting the course instructor. Graders are never permitted to interact with students and are instructed to forward any student communication to the study coordinator. The instructor has the sole discretion to alter any submitted grade. Instructors maintain a record of all grade contests, and any alterations are flagged when rubrics are downloaded at the end of the semester.
Several days after each short writing assignment, students complete a 40-question multiple choice exam covering the same material as the preceding SWA. The final question of each exam asks students to report the number of questions they expect to get correct. Similar to the SWAs, honesty is incentivized by providing extra credit for accuracy.
After the assignment grades have been posted for the first and second assignments, students will be asked to provide a rating of their perceived fairness of the each grade they were assigned (1 to 5 where 1 is “Very Unfair” and 5 is “Very Fair”). This response is not incentivized. On the final exam, students will be asked to provide a rating of their perceived fairness of the overall grading in the course (1 to 5 where 1 is “Very Unfair” and 5 is “Very Fair”).
At the end of the semester, each SWA will be regraded blindly by two additional randomly-assigned graders.

We randomize treatment assignment at the individual level. Randomization by the computer.

Individual participant.

No

3 Short Writing Assignments for each participant, for approximately 7122 observations (expected to change based on drop/adds)

3 Short Writing Assignments for each participant, for approximately 7122 observations (expected to change based on drop/adds)

3 Short Writing Assignments for each participant, for approximately 7122 observations (expected to change based on drop/adds).