Experimental Design Details
We conduct our experiment through the medium of Quiz 2 only, to be conducted on January 27, 2022. The quiz is conducted online over LMS, and comprised of 20 multiple-choice questions to be answered by students in 30 minutes. A bank of questions subdivided into several subtopics is uploaded on LMS, and LMS is programmed to pick a fixed number of questions from each subtopic, ensuring a total of 20 questions. Students can see only one question at a time on their screen, and the software does not allow them to go back to the previous question, after proceeding to the next question. Also, no two students are likely to receive the same set of 20 questions in exactly the same order since these questions are picked by the LMS in random order for each student.
Our sample consists of all students in these 7 sections (609 enrolled - 295 female students and 314 male students), who will be randomly assigned at individual level into the control (No Saliency) and treatment (Saliency) groups. While students in both the control and treatment groups will receive the randomly selected 20-questions on the quiz, control group students will be instructed through the LMS that each question carries 1-mark with a maximum attainable score of 20 marks on the quiz, and students in the saliency treatment will be instructed that each question carries 5-mark with a maximum attainable score of 100 marks on the entire quiz. We will randomize intervention at the individual level, stratified by section and gender. Such stratification will help attain balance on gender which will also allow us to conduct heterogeneity analysis along the dimension of gender. We also have information on students baseline scores and the programs (major) they are pursuing. We will see if the intervention has differential impact along that dimension.
All students are to be sent a uniform email about the course, quiz date and time, with the mention that quiz carries 10% of marks in their final scores. No mention of the marks assigned to the quiz 2 is made in the email. All instructors are also informed to not share any information on the marks allotted to the quiz. This ensure that students have identical information before the quiz and therefore, could not influence their preparation for the quiz. The research design of our experiment thus ensures that any differences observed in student performance between the two groups is due to the saliency of marks, an information revealed to students right at the beginning of the quiz.
The saliency of marks potentially nudges students to adopt different risk-attitudes at attempting the quiz, revising their strategic approach to writing the quiz. As such, the intervention may ultimately impact the effort students exert on the test as reflected in their time taken to attempt the entire quiz (first half and second half of quiz separately too), the number of total questions attempted, and proportion of attempted questions that were answered correctly, and proportion of total questions that were answered correctly.
While it would be good to measure students' risk-attitudes directly, it would be difficult to have such a task keeping in mind the natural field experiment nature of our study. We however plan to elicit a proxy for whether the intervention affects risk-attitudes between the two groups by showing an additional out-of-the-course bonus question at the end of the quiz, to which they have a choice to answer or skip. If they choose to answer, a correct answer will get them additional 1 mark (or 5 mark if in treatment) and an incorrect answer will take away 0.5 mark (or 2.5 if in treatment), subject to their aggregate score not going beyond 20 marks (or 100 marks if in treatment). Students' decision to this question would inform whether the intervention affected the risk-attitudes, thus providing evidence on one potential mechanism driving the treatment effect on student performance.
To ensure that the intervention is salient, we will show the marks allotted to each question and maximum attainable marks on the quiz - both on the instructions page and on each subsequent page showing the questions. The high-stake nature of the quiz prevents any selective absenteeism on the day of quiz.
(Amendment, February 4 2022) After conducting the first leg of the experiment, we have decided to extend our experiment to an additional setting of low-stake quiz too. In order to study that role of saliency of marks in a zero-stake and low intrinsic motivation setting, we will be conducting quiz of a similar nature (20 multiple choice questions and 30 minutes) with students of the 18 sections of communication courses offered at the university. Around 915 students are enrolled in those 18 sections who will become the part of our experiment. In this zero stake setting, students will be offered the incentive of an additional attendance if they attempt the quiz. As with the first experiment, this quiz will be conducted at the same time for all students across all 18 sections, and it will be conducted over LMS.
Similar to the first experiment, we will also have one question at the end of the quiz which attracts a positive score on correct attempt and a negative score in incorrect attempt. This question is to proxy their risk-attitudes, and to see whether saliency of higher marks makes them more risk-averse as expected from prior theoretical literature. We must admit though that given the zero-stake nature of the quiz, it is possible that students may not show any differences in their response to this question. However, if they do, then it would provide evidence that even in the zero-stake settings, saliency of marks can impact risk-attitudes of students, which may be the mechanism affecting their responses to the first 20 questions of the main quiz.
(Amendment, Feb 8 2022) The second leg of the experiment is to be conducted on Feb 9 2022.
One amendment from the first leg of the experiment: We will not have the bonus question intended to proxy their risk-attitudes due to the operational complexity it created in explaining it to the students in the first leg of the experiment, due to the natural field experiment nature of our study.
Additionally, the course coordinator of these courses has informed that some students have another class at the same time as the quiz. This is because one common time slot has been fixed for all enrolled students across 18 sections pursuing the communications courses. We expect an absenteeism rate of about 10-20%. However, this absenteeism is unlikely to be due to the experiment or intervention since this information is not revealed to students (or to instructors, only the course coordinator is aware of the research nature of this study). We will, however, assign all students to a treatment group, and will later do a statistical test to see whether treatment predicted absenteeism.