Nudging in complex environments

Subjects play through a series of modules of a computer game that is about preparing food in a home kitchen setting. The goal of the game is to prepare a dish under hygienic circumstances, subject to time pressure and disturbances. We measure performance in the game based on a number of important food safety actions. Subjects are recruited on the crow working platform Prolific.

2021-11-18

2022-11-30

The number of correctly applied important food safety actions (IFSA).

IFSAS are actions that the decision maker takes at pre-defined check points (when this check point is reached for the first time in a recipe). For each recipe, there are 12 or 13 IFSAs (every 2nd recipe on average has the bread dropping to the floor when a decision maker tries to put it on the cutting board), which are the following:
1. Wash hands at start of recipe
2. Rinse vegetable/fruit
3. Wash hands after raw chicken handled
4. Wash hands after unwashed vegetable/fruit handled
5. Clean the knife after raw chicken cut
6. Clean the cutting board after raw chicken cut
7. Clean the knife after vegetable/fruit cut
8. Clean the cutting board after vegetable/fruit cut
9. Clean worktop after chicken cut
10. Clean worktop after vegetable/fruit cut
11. Check with thermometer if chicken reached 74°C
12. Do not rinse raw chicken
13. Throw out dropped bread (only relevant in recipes where bread drops)

Depending on the hypothesis, we distinguish between actions about which the decision maker is reminded in game module 3 (hand washing in Reminder; hand washing, cleaning the worktop, and checking the meat temperature in ManyReminders) and actions about which the decision maker is not reminded. Next to the overall number of correct actions, we also aggregate the correctly applied actions in the respective categories (reminded vs. non-reminded).

We use secondary outcome variables to assess the robustness of our results, explore the mechanisms behind our results, and to conduct further exploratory data analysis.

Our between-subject experiment consists of two studies with three different conditions each (2 treatments, 1 control) and is run online with UK residents using the Prolific crowd working platform. In all conditions, subjects play through a series of modules of a computer game that is about preparing food in a home kitchen setting.

The study runs online on the Prolific platform. The experiment automatically redirects participants between parts that run on the survey platform Qualtrics and game modules that run as a WebGL app within the browser. At the end of each part of the study, participants are redirected to the Prolific platform to automatically register the completion code.

Date 1 (Part 1). After reading and accepting the consent form, subjects enter game module 1. It starts with a short video introducing the game mechanics. Subjects then play a practice round of the game without time pressure and subsequently complete two recipes under time pressure (a timer appears in the game screen, but the game continues after the 5-minute time limit elapses).

After this module, subjects complete a survey and then receive information about safe food handling in the form of a 2-minute video. The survey consists of questions about food safety related knowledge and behavior targeted by the game. Further, we ask about individual characteristics such as age, household type, risk preferences in general and specific to the food safety domain, food and cooking preferences. The information in the video addresses the following four broad categories: Personal hygiene (such as hand washing), kitchen hygiene (such as cleaning utensils and surfaces), rinsing fresh vegetables and fruits, and handling meat (to not rinse meat; to check with a food thermometer that meat is thoroughly cooked).

Subjects continue with game module 2. It starts with a video showing a play-through of the game that explains correct behavior in the light of the food safety advice that they just received in the video at the end of the survey module. Subjects then play through four recipes to give us data on their behavior after they have received the food safety advice. Subjects do not get any feedback about their performance in the game and do not know yet on what feedback will be given in the remainder of the study.

Subjects then continue with game module 3, which is the stage were the experimental variation of the study happens. Subjects are randomly assigned to one of the three experimental conditions (Control, Reminder, and ManyReminders), stratified by gender to achieve balance across conditions. In all three conditions, subjects are first told that their payment from now on will depend on how well they play the game: Next to the fixed payment (GBP 8 for part 1 and GBP 6 for part 2) there is a bonus of up to GBP 3 based on how well a subject plays the game from now on. Specifically, one of the five game levels that a subject will subsequently play through in the remainder of date 1 and on date 2 will randomly be drawn at the end of the study. The bonus is then calculated as GBP 3 x (percentage score) of the level drawn, where the percentage score assigns a value between 0 and 1 based on how many of the four recipes in the level were completed within the time limit and how many of the important food safety actions (IFSAs) were correctly performed. A brief video at the start of game module 3 explains to the subjects how the score is determined and what feedback they receive at the end of each level about their performance. Subjects then play through the three levels of game module 3. Each level requires completing four recipes.

Date 2 (Part 2). 48 hours after part 1 was opened up on the Prolific platform, part 2 is opened up for those subjects who completed part 1. Here subjects complete game module 4, which is identical across all treatments. Subjects play the game and receive feedback after each of two levels (each level consists of four recipes). That is, the reminders are withdrawn for the Reminder and ManyReminder treatments. After game play, subjects answer a few questions about enjoyment of the game, possible information seeking regarding food safety during the time of the experiment, and they can enter comments about the study before registering their completion of the study.

Additional treatments:
Subjects do not get any feedback on their performance in the game (instead of getting feedback on their performance at the end of each level in modules 3 and 4). Information in the videos is adjusted accordingly.

Randomization (stratified to achieve gender balance across treatments) done using the randomizer feature of Qualtrics.

Individual

No

The treatments are not clustered

We plan for a minimum of 750 individuals who complete both parts of the study. As we expect some attrition between parts 1 and 2 of this two-part study, we plan to initially recruit 1000 individuals. We will continue sampling with random allocation to treatments until one of the following holds: at least 250 individuals completed both parts in every treatment, we exhaust our budget, or we reach January 31, 2022. Additional treatments We again plan for a minimum of 750 individuals who complete both parts of the study. As we expect some attrition between parts 1 and 2 of this two-part study, we plan to initially recruit 1000 individuals. We will continue sampling with random allocation to treatments until one of the following holds: at least 250 individuals completed both parts in every treatment, we exhaust our budget, or we reach November 30, 2022.

A minimum of 250 individuals in Control, a minimum of 250 individuals in Reminder, and a minimum of 250 individuals in ManyReminders.

Additional treatments
A minimum of 250 individuals in each of the three additional treatments.

Power calculations are based on pilot data for the Control condition. Using the program "pc_simulate" by Burlig, Preonas, and Woerman (2017), we conducted simulation-based power calculations in STATA for ANCOVA regressions with 7 pre-treatment periods and 4 post-treatment periods, a Type-I error rate of alpha=0.01 or alpha=0.05, power 0.8, 1000 simulations per effect size, and assuming pairwise treatment comparisons with 250 observations per treatment (i.e. a total sample size of 500). 1. Total number of correct IFSAs per recipe (also referred to as number of correct actions) - Range {0,1,...,12} or {0,1,...,13} depending on the recipe - Values in pilot for Control and Minimum Detectable Effect Size (MDE): * First four recipes of game module 3: Mean 7.19 (std.dev 3.45) MDE (one-sided): 0.33 (alpha=0.01)/ 0.27 (alpha=0.05) MDE (two-sided): 0.35 (alpha=0.01)/ 0.30 (alpha=0.05) * First four recipes of game module 4: Mean 7.89 (std.dev 2.57) MDE (one-sided): 0.28 (alpha=0.01)/ 0.34 (alpha=0.05) MDE (two-sided): 0.35 (alpha=0.01)/ 0.32 (alpha=0.05) 2. The correct IFSAs per recipe for which a reminder is given in the Reminder treatment (handwashing): - Range {0,1,2,3} - Values in pilot for Control and Minimum Detectable Effect Size (MDE): * First four recipes of game module 3: Mean 1.89 (std.dev 0.93) MDE (one-sided): 0.15 (alpha=0.01)/ 0.12 (alpha=0.05) * First four recipes of game module 4: Mean 2.00 (std.dev 0.71) MDE (one-sided): 0.12 (alpha=0.01)/ 0.09 (alpha=0.05) 3. The correct IFSAs per recipe about which no reminders are given in Reminder (i.e. total number of correct IFSAs – correct handwashing IFSAs) - Values in pilot for Control and Minimum Detectable Effect Size (MDE): * First four recipes of game module 3: Mean 5.89 (std.dev 2.42) MDE (two-sided): 0.23 (alpha=0.01)/ 0.20 (alpha=0.05) * First four recipes of game module 4: Mean 5.89 (std.dev 2.15) MDE (two-sided): 0.29 (alpha=0.01)/ 0.24 (alpha=0.05) 4. The correct IFSAs per recipe for which a reminder is given in the ManyReminders treatment (handwashing, cleaning the worktop, and checking the meat temperature): - Range {0,1,…,6} - Values in pilot for Control and Minimum Detectable Effect Size (MDE): * First four recipes of game module 3: Mean 3.44 (std.dev 1.33) MDE (one-sided): 0.23 (alpha=0.01)/ 0.19 (alpha=0.05) * First four recipes of game module 4: Mean 3.33 (std.dev 1.22) MDE (one-sided): 0.17 (alpha=0.01)/ 0.14 (alpha=0.05)