Marks saliency and student performance in a low and high stake exam

Last registered on February 08, 2022

Pre-Trial

Trial Information

General Information

Title
Marks saliency and student performance in a low and high stake exam
RCT ID
AEARCTR-0008818
Initial registration date
January 19, 2022

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
January 19, 2022, 12:24 PM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
February 08, 2022, 2:18 PM EST

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Region

Primary Investigator

Affiliation
Ahmedabad University

Other Primary Investigator(s)

PI Affiliation
Ahmedabad University

Additional Trial Information

Status
In development
Start date
2022-01-17
End date
2022-04-30
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
A student’s performance on a test is highly correlated with the time and effort exerted in preparation prior to the test. But even with similar preparation levels, some outperform others. Among other factors, this may be attributed to students’ differential intrinsic motivation affecting their effort exerted during the test per se. Gneezy et. al (2019) showed that the low intrinsic motivation during a test could be mitigated by offering pecuniary incentives tied to student’s performance on the test. Our paper extends this literature and studies the effectiveness of a subtler zero-cost behavioral nudge in increasing students’ effort during a test. More specifically, we study how varying the saliency of maximum points affects students’ exerted effort and performance during the test. We study this question in two contexts: (a) High stake quiz for microeconomics course, and (b) Zero-stake aptitude test conducted for students persuing communication course at the university.
External Link(s)

Registration Citation

Citation
Arora, Puneet and Ishita Tripathi. 2022. "Marks saliency and student performance in a low and high stake exam." AEA RCT Registry. February 08. https://doi.org/10.1257/rct.8818
Experimental Details

Interventions

Intervention(s)
The first leg of the experiment is conducted in principles of microeconomics course offered at a large private university in India. The course is part of a general education curriculum. It is the first course in economics offered to new students across the university, with most students having a limited choice over enrollment in the course. Students in one of the high-stake assessments are divided into control and treatment where they take the same quiz with different scale of marks assigned to it.

The second leg of the experiment is conducted in communications course offered at the same university. Experimental details stay same as in the first leg of the experiment, except that the quiz here is of zero-stake for them, and is a multiple choice aptitude test comprising of some math, some logical reasoning and some english comprehension questions. The only incentive for students to attempt this quiz is one additional attendance that they will receive in their communication class on attempting this quiz. Thus, this is a low stake and low intrinsic motivation context as against the first leg of the experiment which was a high stake and high intrinsic motivation context.
Intervention Start Date
2022-02-12
Intervention End Date
2022-02-28

Primary Outcomes

Primary Outcomes (end points)
1. No. of Correct questions (total marks)
2. No. of attempted questions
3. Proportion of attempted questions that are correct questions

Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
1. Total time spent answering the quiz
2. Time spent in answering first 10 questions
3. Time spent in answering last 10 questions
4. Proportion of students who attempted the bonus question (risk-taking)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
The first leg of the experiment is conducted in principles of microeconomics course offered at a large private university in India. The course is part of a general education curriculum. It is the first course in economics offered to new students across the university, with most students having a limited choice over enrollment in the course. The course is offered through 7 sections (5 instructors; 3 females and 2 males) in a bi-semester period that started in October 2021 and ended in February 2022. Students' assessment in the course occurrs using 3 quizzes, 1 group project, and 1 final exam which are all identical across the 7 sections. The teaching and assessments are all conducted in English language, which is the official language of instruction at the university. Students in one of the high-stake assessments are divided into control and treatment where they take the same quiz with different scale of marks assigned to it.

The same experiment will also be conducted in a low-stake setting with students pursuing communications course. There are 18 sections of this course offered by multiple instructors. They will write a zero-stake quiz for which the only incentive they have to attempt is an additional attendance that they will receive to attempt the quiz.

We must note that students who are pursuing both the courses simultaneously will be a part of both the experiments. We will control for that in our statistical analysis by adding a dummy for students who became part of both experiments to see if they behaved any differently from others. After the second leg of the experiment, students in both legs of the experiment will be briefed about the nature of the study to request their consent.
Experimental Design Details
We conduct our experiment through the medium of Quiz 2 only, to be conducted on January 27, 2022. The quiz is conducted online over LMS, and comprised of 20 multiple-choice questions to be answered by students in 30 minutes. A bank of questions subdivided into several subtopics is uploaded on LMS, and LMS is programmed to pick a fixed number of questions from each subtopic, ensuring a total of 20 questions. Students can see only one question at a time on their screen, and the software does not allow them to go back to the previous question, after proceeding to the next question. Also, no two students are likely to receive the same set of 20 questions in exactly the same order since these questions are picked by the LMS in random order for each student.

Our sample consists of all students in these 7 sections (609 enrolled - 295 female students and 314 male students), who will be randomly assigned at individual level into the control (No Saliency) and treatment (Saliency) groups. While students in both the control and treatment groups will receive the randomly selected 20-questions on the quiz, control group students will be instructed through the LMS that each question carries 1-mark with a maximum attainable score of 20 marks on the quiz, and students in the saliency treatment will be instructed that each question carries 5-mark with a maximum attainable score of 100 marks on the entire quiz. We will randomize intervention at the individual level, stratified by section and gender. Such stratification will help attain balance on gender which will also allow us to conduct heterogeneity analysis along the dimension of gender. We also have information on students baseline scores and the programs (major) they are pursuing. We will see if the intervention has differential impact along that dimension.

All students are to be sent a uniform email about the course, quiz date and time, with the mention that quiz carries 10% of marks in their final scores. No mention of the marks assigned to the quiz 2 is made in the email. All instructors are also informed to not share any information on the marks allotted to the quiz. This ensure that students have identical information before the quiz and therefore, could not influence their preparation for the quiz. The research design of our experiment thus ensures that any differences observed in student performance between the two groups is due to the saliency of marks, an information revealed to students right at the beginning of the quiz.

The saliency of marks potentially nudges students to adopt different risk-attitudes at attempting the quiz, revising their strategic approach to writing the quiz. As such, the intervention may ultimately impact the effort students exert on the test as reflected in their time taken to attempt the entire quiz (first half and second half of quiz separately too), the number of total questions attempted, and proportion of attempted questions that were answered correctly, and proportion of total questions that were answered correctly.

While it would be good to measure students' risk-attitudes directly, it would be difficult to have such a task keeping in mind the natural field experiment nature of our study. We however plan to elicit a proxy for whether the intervention affects risk-attitudes between the two groups by showing an additional out-of-the-course bonus question at the end of the quiz, to which they have a choice to answer or skip. If they choose to answer, a correct answer will get them additional 1 mark (or 5 mark if in treatment) and an incorrect answer will take away 0.5 mark (or 2.5 if in treatment), subject to their aggregate score not going beyond 20 marks (or 100 marks if in treatment). Students' decision to this question would inform whether the intervention affected the risk-attitudes, thus providing evidence on one potential mechanism driving the treatment effect on student performance.

To ensure that the intervention is salient, we will show the marks allotted to each question and maximum attainable marks on the quiz - both on the instructions page and on each subsequent page showing the questions. The high-stake nature of the quiz prevents any selective absenteeism on the day of quiz.


(Amendment, February 4 2022) After conducting the first leg of the experiment, we have decided to extend our experiment to an additional setting of low-stake quiz too. In order to study that role of saliency of marks in a zero-stake and low intrinsic motivation setting, we will be conducting quiz of a similar nature (20 multiple choice questions and 30 minutes) with students of the 18 sections of communication courses offered at the university. Around 915 students are enrolled in those 18 sections who will become the part of our experiment. In this zero stake setting, students will be offered the incentive of an additional attendance if they attempt the quiz. As with the first experiment, this quiz will be conducted at the same time for all students across all 18 sections, and it will be conducted over LMS.

Similar to the first experiment, we will also have one question at the end of the quiz which attracts a positive score on correct attempt and a negative score in incorrect attempt. This question is to proxy their risk-attitudes, and to see whether saliency of higher marks makes them more risk-averse as expected from prior theoretical literature. We must admit though that given the zero-stake nature of the quiz, it is possible that students may not show any differences in their response to this question. However, if they do, then it would provide evidence that even in the zero-stake settings, saliency of marks can impact risk-attitudes of students, which may be the mechanism affecting their responses to the first 20 questions of the main quiz.

(Amendment, Feb 8 2022) The second leg of the experiment is to be conducted on Feb 9 2022.
One amendment from the first leg of the experiment: We will not have the bonus question intended to proxy their risk-attitudes due to the operational complexity it created in explaining it to the students in the first leg of the experiment, due to the natural field experiment nature of our study.

Additionally, the course coordinator of these courses has informed that some students have another class at the same time as the quiz. This is because one common time slot has been fixed for all enrolled students across 18 sections pursuing the communications courses. We expect an absenteeism rate of about 10-20%. However, this absenteeism is unlikely to be due to the experiment or intervention since this information is not revealed to students (or to instructors, only the course coordinator is aware of the research nature of this study). We will, however, assign all students to a treatment group, and will later do a statistical test to see whether treatment predicted absenteeism.
Randomization Method
Randomization in both legs of the experiment done on computer before the experiment (randtreat command used in stata to randomize, stratified by section and gender)
Randomization Unit
Randomization in both legs of the experiment is conducted at individual level - stratified by section and gender
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
Leg 1 (High-stake quiz): 609 students (in 7 microeconomics sections)
Leg 2 (Zero-stake quiz): 915 students (in 18 communication course sections)
Sample size: planned number of observations
609 + 915 = 1524
Sample size (or number of clusters) by treatment arms
304 students control and 305 students treatment in leg 1 (high-stake)
457 students control and 458 students treatment in leg 2 (zero-stake)
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Students' score on previous quiz had a mean score of 10.6 with a standard deviation of 3.25. Assuming this distribution of scores to also follow in Quiz 2, we have 86% statistical power to detect an effect size of 0.4 marks with our sample size of 609 students.
IRB

Institutional Review Boards (IRBs)

IRB Name
Ahmedabad University Research Board
IRB Approval Date
2022-02-01
IRB Approval Number
N/A

Post-Trial

Post Trial Information

Study Withdrawal

There are documents in this trial unavailable to the public. Use the button below to request access to this information.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials