Experimental Design Details
We first run a pilot where we test our proposed level of allowed mistakes before losing experimental earnings. We also test whether the level of incentives have an effect on performance.
The pilot will consist of two work parts and a questionnaire.
The questionnaire will be divided in two parts, a pre-survey demographic questionnaire which is filled out as students sign up for the experiment and a post-survey questionnaire. In these questionnaires several factors in our post- experiment survey: exam preparation, subject ability, motivation in the task, and anxiety in the task.
In the work parts, participants are asked to solve seven mathematical tasks in five minutes. The questions are similar to the Raven's test, where the participants see a sequence of images and must complete the pattern by choosing one of five alternatives. After choosing an option, participatns answer questions about the confidence in their answers and whether they guessed or not.
During both part one and part two, a clock with the remaining time is visible on participants' screens. To simulate the real examination, participants also have the possibility of skipping questions and return to them at the end of the test. At the end of the part they see a screen with thumbnails of each question and they can click on the thumbnail to return to the questions they skipped in turn. After each submitted response participants are taken to a new page where the timer is stopped - and they have twenty seconds to answer how confident they feel that their submitted answer was correct. If the timer runs out and participants fail to answer any questions, their response is coded as wrong. There is no punishment for wrong answers.
After the first work part, the participants face one of the treatment conditions which is explained in detail below. After finishing this, they answer a post-survey questionnaire.
Near versus far from cutoff
In the near the cutoff condition, if the participants get two or more questions wrong, they obtain a payoff of zero. That is, they can only get maximum two questions wrong before they "miss the cutoff". In the far from cutoff condition, they can get up to 5 questions wrong before they come below the cutoff. In this way we are able to study whether the cost of making a mistake has an effect on the performance of the participants. Participants do not know whether have answered correctly until the test is completed.
Incentives
One main concern is that the above treatments may induce different behaviour as they have different expected payoffs. We want to test if any behavioural differences we see empirically is driven incentive levels, and not by our the pressure from the mistakes. We propose to test for incentive concerns in the following way:
Incentive manipulation 1
First, we address the fact that in the treatments where mistakes are costly, the expected payoff for each participant is lower than in those treatments where mistakes are less costly. To address these concerns, we introduce one treatment where we increase the bonus payment in the "near-cutoff"-condition such that the expected payoff for the "far-from-cutoff"-condition is equal to the "near-cutoff"-condition. The calculations for this is shown in the attached pre-analysis plan.
Incentive manipulation 2
Second, we add a treatment where participants are only paid their experimental earnings with a ten percent probability. This is to address concerns that the stakes will be too low in the main survey experiment. We will not have the funding to pay every participant their full experimental earnings in the main survey experiment so this is to test any behavioural effects on a similar sample to the main experiment sample.