Nudging in complex environments

Subjects play through a series of modules of a computer game that is about preparing food in a home kitchen setting. The goal of the game is to prepare a dish under hygienic circumstances, subject to time pressure and disturbances. We measure performance in the game based on a number of important food safety actions. Subjects are recruited on the crow working platform Prolific.

The study lets subjects play through four game modules. Treatment variation occurs in game module 3 only. Game modules 1, 2, and 4 are identical across treatments (see experimental design for details).

Treatments:
Control: Subjects play the game. In game modules 3 and 4, they receive feedback about important food safety actions (IFSAs) after four recipes are played.
Reminder: Subjects receive reminders about one type of IFSA in game module 3, namely whenever they are supposed to wash hands.
ManyReminders: Subjects receive reminders about three types of IFSAs in game module 3 (washing hands, cleaning surfaces, and checking the temperature of the meat with a food thermometer).

Additional treatments:
We run the same treatments as above with the following differences:
- Subjects do not get any feedback on their performance in the game (instead of getting feedback on their performance at the end of each level in modules 3 and 4).
- Subjects play only one level in module 4 (instead of two).
We otherwise follow the same procedures and test the same hypotheses as in the original treatments.

Hypotheses
Overall, our hypotheses concern the effects of reminders (i) on immediate actions in game module 3 and (ii) on habit formation, by looking at actions in game module 4 where there no longer are reminders. The tests rely on comparing the number of important food safety actions (IFSAs) performed in the relevant modules between treatment(s) and control. To test for spillover effects, we distinguish in these analyses between reminded and non-reminded actions.

We test a total of five hypotheses. We also outline some secondary analyses that provide insights into possible mechanisms behind the main results. These secondary analyses are not part of the power analysis and may not be performed if the results are not confirmed. Further tests of mechanisms may be added as the analysis evolves.

Hypothesis 1:
In the first four recipes of game module 3, the decision makers perform more correct actions in Reminder than in Control. We disentangle the effect into:
(i) Direct effects of the reminders: The decision makers perform more often the correct actions about which they are reminded in Reminder than in Control.
(ii) Indirect effects of the reminders: The decision makers perform more or less often the correct actions about which they receive no reminders in Reminder than in Control.

Explanation of Hypothesis 1.
The first hypothesis is that reminders have an overall net positive effect on the total number of correct actions. Based on the literature, we expect that reminders help the decision makers to more often perform the correct actions about which they are reminded, leading to Hypothesis 1 (i). Yet, the effect of reminders on the actions about which subjects receive no reminders in game module 3 in Reminder is less clear. The literature has mainly focused on negative spillovers from nudges. In our setting, reminders also may have negative spillover effects on other actions (crowding out). First, reminders may increase cognitive load or be a distraction. Second, performing the reminded actions may lead subjects to compensate by performing fewer of the other actions (licensing). The novel feature of our setting is that it allows us to test for two types of theoretically predicted positive spillover effects. First, the reminder may freeze attention: Because subjects know that they will be reminded of hand washing, this potentially frees up attention resources that decision makers can direct towards the non-reminded actions. Second, the reminder may direct attention also to complementary actions that the decision maker associates with the reminded action. Whether negative or positive spillover effects dominate hence is an open empirical question that is central in our paper. That is why Hypothesis 1 (ii) is not directional.

To better understand the results for Hypothesis 1, we subsequently test for specific spillover effects. Specifically, we want to identify on which (sets of) actions the reminder has a positive spillover effect and on which actions it has a negative spillover effect. For example, a reminder is more likely to have a positive spillover effect on complementary actions or on actions which are associated with hand washing than on non-similar actions (e.g., washing the cutting board and knife are actions similar to hand washing, whereas cleaning surfaces is less similar to hand washing). Further, it seems plausible that crowding-out is more likely to occur for actions that take longer (such as cleaning the surface, washing the cutting board and knife, throwing-out and replacing the bread) compared to actions that take less time (such as rinsing vegetables or checking the meat temperature). Each action has the same impact on the percentage score and thus carries the same monetary reward. But because actions take different amounts of time to perform, the relative incentives to take an action may differ. Similarly, we may expect crowding-out of the non-rewarded actions (removing the cat, cleaning after the cat). Finally, the dropped bread comes with a strong visual cue to throw it out (cat hair) which could induce more compliance in general than with the other (non-reminded) IFSAs for which such a strong cue is not given. This IFSA hence is an indicator of compliance with a visual cue to perform a correct action that is available in all three treatments. We can use it to investigate in exploratory analyses the impact of reminders only on the subsample who are ‘responsive’ according to this indicator of compliance.

Hypothesis 2:
In the first four recipes of game module 3, the decision makers perform more or less often the correct actions in ManyReminders than in Reminder.

Explanation of Hypothesis 2.
The second hypothesis concerns the net effect that multiple reminder nudges have on behavior. On the one hand, many reminders may lead to cognitive overload and thus to crowding-out of attention. On the other hand, many reminders may also help the decision maker if they still consistently prompt the decision maker to perform the reminded actions and free up attention resources that can be devoted to remembering to perform the non-reminded actions. Hence, it is an empirical question whether more reminders have a net positive effect on the total number of correct actions compared to reminders about a single type of action. That is why Hypothesis 2 is not directional.

To understand the underlying mechanisms, we look separately at the effects of the different treatments on the actions that are not reminded in any treatment, on hand washing (the reminded action in both Reminder and ManyReminders), and on cleaning the worktop and checking the meat temperature (the two additional types of reminded actions in ManyReminders). That is, for each of the following three sets of actions, we test whether the decision maker performs the respective actions more or less often in game module 3 in ManyReminders than in Reminder: (i) actions that are not reminded in any treatment, (ii) hand washing (in general; and specifically, after washing meat and vegetables) and (iii) cleaning the surfaces and checking the meat temperature.

To investigate possible negative effects of multiple reminders, we test in addition whether showing multiple reminders at the same time on the screen leads to worse performance on a particular reminded action than just having a single reminder on the screen about this action. Specifically, after handling raw meat, three reminders are given at the same time in ManyReminders. This may crowd out attention of the decision maker to hand washing compared to a decision maker in Reminder who receives just a hand washing reminder at this stage. At the start of the game and after handling unwashed vegetables or fruit, the decision maker receives a single reminder on the screen in both treatments, so we expect no treatment differences in behavior here.

If we do not find a significant difference between Reminder and ManyReminders, we will pool the two treatments to have more observations for the comparison with Control.

Hypothesis 3:
In the first four recipes of game module 4, the decision makers more often perform the correct actions about which they got reminded in game module 3 in Reminder than decision makers in Control do.

Explanation of Hypothesis 3.
Since we expect a positive effect of reminders on the actions about which decision makers are reminded in game module 3 (cf. Hypothesis 1(i)), decision makers in Reminder will be more likely to form hand-washing habits than decision makers in Control, that they then maintain when the reminders are withdrawn in game module 4. This leads to Hypothesis 3.

If Hypothesis 3 is not rejected, we will test whether reminders also have a net positive impact when considering also the non-reminded actions.

Hypothesis 4:
In the first four recipes of game module 4, the decision makers more often perform the correct actions in Reminder than decision makers in Control do.

Hypothesis 5:
In the first four recipes of game module 4, the decision makers perform more/less often (with the same direction as in the empirical result for Hypothesis 2) the correct actions in ManyReminders than in Reminder.


Explanation of Hypothesis 5.
The hypothesis addresses the novel question whether it is possible to scale up the nudging of habit formation through reminders on multiple types of actions.


Secondary hypotheses:

For all hypotheses comparing Reminder vs. Control, we test the analogous hypotheses for ManyReminders vs Control.

All our hypotheses are based on the recipes completed before feedback is given in a game module. We also test how feedback affects the hypotheses, by considering the recipes after feedback is given in a game module.

Finally, if the hypotheses are not rejected, we will explore how Reminder and ManyReminders compare to Control on the non-reminded actions.

2021-11-18

2022-11-30

The number of correctly applied important food safety actions (IFSA).

IFSAS are actions that the decision maker takes at pre-defined check points (when this check point is reached for the first time in a recipe). For each recipe, there are 12 or 13 IFSAs (every 2nd recipe on average has the bread dropping to the floor when a decision maker tries to put it on the cutting board), which are the following:
1. Wash hands at start of recipe
2. Rinse vegetable/fruit
3. Wash hands after raw chicken handled
4. Wash hands after unwashed vegetable/fruit handled
5. Clean the knife after raw chicken cut
6. Clean the cutting board after raw chicken cut
7. Clean the knife after vegetable/fruit cut
8. Clean the cutting board after vegetable/fruit cut
9. Clean worktop after chicken cut
10. Clean worktop after vegetable/fruit cut
11. Check with thermometer if chicken reached 74°C
12. Do not rinse raw chicken
13. Throw out dropped bread (only relevant in recipes where bread drops)

Depending on the hypothesis, we distinguish between actions about which the decision maker is reminded in game module 3 (hand washing in Reminder; hand washing, cleaning the worktop, and checking the meat temperature in ManyReminders) and actions about which the decision maker is not reminded. Next to the overall number of correct actions, we also aggregate the correctly applied actions in the respective categories (reminded vs. non-reminded).

We use secondary outcome variables to assess the robustness of our results, explore the mechanisms behind our results, and to conduct further exploratory data analysis.

Our between-subject experiment consists of two studies with three different conditions each (2 treatments, 1 control) and is run online with UK residents using the Prolific crowd working platform. In all conditions, subjects play through a series of modules of a computer game that is about preparing food in a home kitchen setting.

The study runs online on the Prolific platform. The experiment automatically redirects participants between parts that run on the survey platform Qualtrics and game modules that run as a WebGL app within the browser. At the end of each part of the study, participants are redirected to the Prolific platform to automatically register the completion code.

Date 1 (Part 1). After reading and accepting the consent form, subjects enter game module 1. It starts with a short video introducing the game mechanics. Subjects then play a practice round of the game without time pressure and subsequently complete two recipes under time pressure (a timer appears in the game screen, but the game continues after the 5-minute time limit elapses).

After this module, subjects complete a survey and then receive information about safe food handling in the form of a 2-minute video. The survey consists of questions about food safety related knowledge and behavior targeted by the game. Further, we ask about individual characteristics such as age, household type, risk preferences in general and specific to the food safety domain, food and cooking preferences. The information in the video addresses the following four broad categories: Personal hygiene (such as hand washing), kitchen hygiene (such as cleaning utensils and surfaces), rinsing fresh vegetables and fruits, and handling meat (to not rinse meat; to check with a food thermometer that meat is thoroughly cooked).

Subjects continue with game module 2. It starts with a video showing a play-through of the game that explains correct behavior in the light of the food safety advice that they just received in the video at the end of the survey module. Subjects then play through four recipes to give us data on their behavior after they have received the food safety advice. Subjects do not get any feedback about their performance in the game and do not know yet on what feedback will be given in the remainder of the study.

Subjects then continue with game module 3, which is the stage were the experimental variation of the study happens. Subjects are randomly assigned to one of the three experimental conditions (Control, Reminder, and ManyReminders), stratified by gender to achieve balance across conditions. In all three conditions, subjects are first told that their payment from now on will depend on how well they play the game: Next to the fixed payment (GBP 8 for part 1 and GBP 6 for part 2) there is a bonus of up to GBP 3 based on how well a subject plays the game from now on. Specifically, one of the five game levels that a subject will subsequently play through in the remainder of date 1 and on date 2 will randomly be drawn at the end of the study. The bonus is then calculated as GBP 3 x (percentage score) of the level drawn, where the percentage score assigns a value between 0 and 1 based on how many of the four recipes in the level were completed within the time limit and how many of the important food safety actions (IFSAs) were correctly performed. A brief video at the start of game module 3 explains to the subjects how the score is determined and what feedback they receive at the end of each level about their performance. Subjects then play through the three levels of game module 3. Each level requires completing four recipes.

Date 2 (Part 2). 48 hours after part 1 was opened up on the Prolific platform, part 2 is opened up for those subjects who completed part 1. Here subjects complete game module 4, which is identical across all treatments. Subjects play the game and receive feedback after each of two levels (each level consists of four recipes). That is, the reminders are withdrawn for the Reminder and ManyReminder treatments. After game play, subjects answer a few questions about enjoyment of the game, possible information seeking regarding food safety during the time of the experiment, and they can enter comments about the study before registering their completion of the study.

Additional treatments:
Subjects do not get any feedback on their performance in the game (instead of getting feedback on their performance at the end of each level in modules 3 and 4). Information in the videos is adjusted accordingly.

Randomization (stratified to achieve gender balance across treatments) done using the randomizer feature of Qualtrics.

Individual

No

The treatments are not clustered

We plan for a minimum of 750 individuals who complete both parts of the study. As we expect some attrition between parts 1 and 2 of this two-part study, we plan to initially recruit 1000 individuals. We will continue sampling with random allocation to treatments until one of the following holds: at least 250 individuals completed both parts in every treatment, we exhaust our budget, or we reach January 31, 2022. Additional treatments We again plan for a minimum of 750 individuals who complete both parts of the study. As we expect some attrition between parts 1 and 2 of this two-part study, we plan to initially recruit 1000 individuals. We will continue sampling with random allocation to treatments until one of the following holds: at least 250 individuals completed both parts in every treatment, we exhaust our budget, or we reach November 30, 2022.

A minimum of 250 individuals in Control, a minimum of 250 individuals in Reminder, and a minimum of 250 individuals in ManyReminders.

Additional treatments
A minimum of 250 individuals in each of the three additional treatments.

Power calculations are based on pilot data for the Control condition. Using the program "pc_simulate" by Burlig, Preonas, and Woerman (2017), we conducted simulation-based power calculations in STATA for ANCOVA regressions with 7 pre-treatment periods and 4 post-treatment periods, a Type-I error rate of alpha=0.01 or alpha=0.05, power 0.8, 1000 simulations per effect size, and assuming pairwise treatment comparisons with 250 observations per treatment (i.e. a total sample size of 500). 1. Total number of correct IFSAs per recipe (also referred to as number of correct actions) - Range {0,1,...,12} or {0,1,...,13} depending on the recipe - Values in pilot for Control and Minimum Detectable Effect Size (MDE): * First four recipes of game module 3: Mean 7.19 (std.dev 3.45) MDE (one-sided): 0.33 (alpha=0.01)/ 0.27 (alpha=0.05) MDE (two-sided): 0.35 (alpha=0.01)/ 0.30 (alpha=0.05) * First four recipes of game module 4: Mean 7.89 (std.dev 2.57) MDE (one-sided): 0.28 (alpha=0.01)/ 0.34 (alpha=0.05) MDE (two-sided): 0.35 (alpha=0.01)/ 0.32 (alpha=0.05) 2. The correct IFSAs per recipe for which a reminder is given in the Reminder treatment (handwashing): - Range {0,1,2,3} - Values in pilot for Control and Minimum Detectable Effect Size (MDE): * First four recipes of game module 3: Mean 1.89 (std.dev 0.93) MDE (one-sided): 0.15 (alpha=0.01)/ 0.12 (alpha=0.05) * First four recipes of game module 4: Mean 2.00 (std.dev 0.71) MDE (one-sided): 0.12 (alpha=0.01)/ 0.09 (alpha=0.05) 3. The correct IFSAs per recipe about which no reminders are given in Reminder (i.e. total number of correct IFSAs – correct handwashing IFSAs) - Values in pilot for Control and Minimum Detectable Effect Size (MDE): * First four recipes of game module 3: Mean 5.89 (std.dev 2.42) MDE (two-sided): 0.23 (alpha=0.01)/ 0.20 (alpha=0.05) * First four recipes of game module 4: Mean 5.89 (std.dev 2.15) MDE (two-sided): 0.29 (alpha=0.01)/ 0.24 (alpha=0.05) 4. The correct IFSAs per recipe for which a reminder is given in the ManyReminders treatment (handwashing, cleaning the worktop, and checking the meat temperature): - Range {0,1,…,6} - Values in pilot for Control and Minimum Detectable Effect Size (MDE): * First four recipes of game module 3: Mean 3.44 (std.dev 1.33) MDE (one-sided): 0.23 (alpha=0.01)/ 0.19 (alpha=0.05) * First four recipes of game module 4: Mean 3.33 (std.dev 1.22) MDE (one-sided): 0.17 (alpha=0.01)/ 0.14 (alpha=0.05)