How messages of encouragement influence students’ grades on exams?

Our intervention takes the form of a message of encouragement that students will receive before their exam. The student will receive the message of encouragement via two channels: e-mail and text message on their mobile device.

Our intervention takes the form of a message of encouragement that students will receive before their exam. The student will receive the message of encouragement via two channels: e-mail and text message on their mobile device.
The English translation of the Hungarian sentence that students in the treated group will receive is the following: “Dear Student! The fact that you will take your exam soon proves that you already have many successful exams behind you! I truly hope that you will succeed in the next one as well, and I wish you every success! Please follow this link and answer three simple questions before your next exam. We will distribute vouchers worth a total of HUF 100,000 that can be redeemed at the SZTE Gift Shop among the respondents. Winners will be notified via email. In the name of Head of the Directorate of Education, Péter Szakál.”
Students in the control group will receive the following e-mail: “Dear Student! Please follow this link to answer three simple questions before your next exam. We will distribute vouchers worth a total of HUF 100,000 that can be redeemed at the SZTE Gift Shop among the respondents. Winners will be notified via email. In the name of Head of the Directorate of Education .”
Students will receive the e-mail message 24 hours before the exam. In sum, in the e-mail communication the treated and control students differ only in the content of the message they will receive since every student (including those in the control condition) will receive a message from the rector which serves the aim of minimizing the Hawthorne effect, since in prior semesters students have not received any messages from the university before the exam. We will have information about whether students have received the e-mail (due to email delivery failure messages). Almost everybody (99.9% of students) has a valid e-mail address.
In addition to the e-mail message, treated students will receive an additional text message on their mobile device before their exam. The English translation of the Hungarian sentence that students in the treated group will receive as a text message on their mobile device is the following: “We wish you good luck in your next exam since, during your educational career, you have already successfully proved your aptitude! SZTE Education Directorate” Students in the control group will not receive text messages on their mobile devices.
Students gave their phone numbers to the university after matriculation. This list of phone numbers is not a supervised list. Approximately 40% of students gave their phone numbers to the university. We will have information about whether their phones have received the text message, but no information about whether the particular phone numbers still belong to the respective students.

2019-12-09

2020-01-31

The primary outcome variable is students’ grades on the exam, which are integers between 1 and 5.

The primary outcome variable is students’ grades on the exam, which are integers between 1 and 5. In the Hungarian system, grade 1 means fail; other grades are equivalent to passing the exam.

In addition to the primary outcome variable, we pre-register secondary outcome variables which will comprise 1 if a student fails the exam (grade 1) and 0 if the student receives any other grades.

We have designed a crossover randomized field experiment (Brown 1980) in which students function as their own control since we have repeated measurements (first and second exam) about the same students.

We have designed a crossover randomized field experiment (Brown 1980) in which students function as their own control since we have repeated measurements (first and second exam) about the same students.
All students from the target population will receive the treatment for ethical reasons. We will randomize, however, the time when students receive the treatment. Based on the value of a random number, we will create two groups. Group A will receive the treatment before their first exam, and will not receive the treatment before their second exam. Group B will receive the treatment before their second exam, but not before their first exam. Students who receive the treatment (irrespective of whether they receive it before the first or the second exam) will be referred to as the treatment group. Students who do not receive the treatment (irrespective of when) will be referred to as the control group.

We will randomize students at the student level. We will employ pair-matched randomization (Imai, King, and Nall 2009). We will first sort the data file according to the following baseline variables: the study program on which the student is enrolled, the level of training, the type of training, the financial form of training, students’ gender, and students’ ability. In the sorted data file, we will identify the most similar pairs of students. We will randomly assign students from the same pair to Group A or Group B based on the value of a randomly generated number.

student

No

No clusters

We expect a sample size of at least 10,000 students in the analytical sample

We have 16,992 different students at the university. Half of them will be treated before the first exam. The other half will be treated before the second exam.

We expect a sample size of at least 10,000 students in the analytical sample. The power calculation obtained with the Optimal Design program yields an effect size of 0.05 at that sample size. Thus, our sample size is large enough to yield a small effect.

szte_pap_x0v_rew.docx