The Influence of Nudging on Summer School Attendance Intentions: A RCT in Flanders

Last registered on September 17, 2024

Pre-Trial

Trial Information

General Information

Title
The Influence of Nudging on Summer School Attendance Intentions: A RCT in Flanders
RCT ID
AEARCTR-0014302
Initial registration date
September 11, 2024

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
September 17, 2024, 11:42 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
September 17, 2024, 2:17 PM EDT

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Region

Primary Investigator

Affiliation
KU Leuven

Other Primary Investigator(s)

PI Affiliation
KU Leuven

Additional Trial Information

Status
Completed
Start date
2024-06-14
End date
2024-06-29
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
In a post-COVID-19 world, in which societies are moving to promote the recovery of learning lost during the pandemic crisis, it becomes increasingly critical to find efficient and cost-effective ways to invest in education systems. Not only that, but the reduced attention given to COVID-19, regardless of its lingering effects, also precipitates a continuous push to fight against these deficits. It is both in this setting, in an acknowledgement of the current length of summer schools in the Flemish region of Belgium, and in a context of strengthened interest in summer schools by the government of Flanders that we implemented our Randomized Control Trial. This research project aims at understanding how a behavioral nudge can help more disadvantaged students to want to participate in Summer Schools, so as to promote the compensation for the learning losses associated with both summer breaks and COVID-19. For this purpose, we intend to use tailored tests and informational prompts with images, presented to primary school students in the Flemish region of Belgium, such that one group is exposed to no information, another with topics on the benefits of school, another on summer schools, a fourth group on a mix of the last two topics, and a fifth on information about how school can help them improve their socio-emotional state. Hence, we can determine the effect of the additional information on their intention to attend summer school, while controlling for cognitive ability through tailored testing before and after the video. Moreover, we can use information provided by participants to detect students coming from more disadvantaged background in order to explore how our prompts and images have impacted these individuals in particular. Simultaneously, using a similar approach, we intend to assess how students' opinions about school can bias their wish to attend summer schools, and how providing them with information contradictory to their beliefs about school can compensate for the effect of this possible bias, as well as perform test for susceptibility of displaying Anchoring Bias (based on definition by Tversky and Kahneman, 1974) in their decision making. This analysis will be done together with an analysis of data about the results of large online campaigns to promote summer schools. This project will provide valuable information concerning the effect of information distribution on the intention to attend summer schools, behavioral biases, as well as a cost-effectiveness analysis of summer schools and their marketing materials so that policymakers can make better education policies.
External Link(s)

Registration Citation

Citation
De Witte, Kristof and Diogo Vieira Nunes da Conceição. 2024. "The Influence of Nudging on Summer School Attendance Intentions: A RCT in Flanders." AEA RCT Registry. September 17. https://doi.org/10.1257/rct.14302-1.1
Experimental Details

Interventions

Intervention(s)
For the Anchoring Bias experiment, students were randomly shown one of the following questions:
- Control: Did you have breakfast today?
- Low Anchor: Do you think you can remember more than 2 pairs of words?
- High Anchor: Do you think you can remember more than 12 pairs of words?

For the Summer School experiment, students were randomly shown one of the following pieces of information:
- Control: No information. These students were only shown a sentence asking them to pay attention to the following questions (i.e. the questions intended to measure their attention to treatment).
- Treatment 1: School is good. These students will be exposed to information on how school can help students learn better and how schools can be beneficial to them, as well as to a set of statements that shed a positive light on school.
- Treatment 2: Information about Summer School. These students will be exposed to information about what are summer schools, what kinds of activities take place in them, the benefits of participating in summer schools, as well as some examples of alternative ways of spending their summer.
- Treatment 3: Both treatments 1 and 2, in this order.
- Treatment 4: Socio-Emotional promotion. These students will show a series of socio-emotional statements tied to an explanation of how schools and summer schools can help develop those skills.
Intervention (Hidden)
Intervention Start Date
2024-06-14
Intervention End Date
2024-06-29

Primary Outcomes

Primary Outcomes (end points)
(1) Intention to participate in a summer school.
(2) Number of pairs of words the students report to be able to remember.
(3) Opinion about school.
Primary Outcomes (explanation)
(1) Intention to participate in a summer school in the summer break of the current academic year is assessed as the degree of agreement with the following statement: "I would like to participate in a summer school." Assessed in a 5-Point Likert Scale (1 = Certainly not; 2 = Rather not; 3 = Don't know yet; 4 = Yes, not yet registered; 5 = Yes, already registered).
(2) Anchoring bias is assessed by seeing differences in the the response to this question between treatment groups. Given the list of words shown to the students, they were asked "How many pairs of words do you think you can remember?" To answer this question, they were supposed to input a number between 0 and 15 (total number of pairs of words), either in number or text format.
(3) This set of questions is intended to understand the sense of belonging and general opinion of the pupil towards their school. In particular, they aim to show whether students have a negative opinion about school (i.e. associate it to a place where they fail and they are punished). Assessed through the agreement with a set of statements in a 5-Point Likert Scale (1 = Totally disagree; 2 = Disagree; 3 = Neither agree nor disagree; 4 = Agree; 5 = Totally agree).

References:
(2) Geurten, Marie and Thierry Meulemans (Feb. 2017). “The effect of feedback on children’s metacognitive judgments: a heuristic account”. In: Journal of Cognitive Psychology 29 (2), pp. 184–201. ISSN: 2044-5911. DOI: 10.1080/20445911.2016.1229669.
(3) SiBO: https://ppw.kuleuven.be/onderwijskunde/en/projects/longitudinal-studies-students-school-careers

Secondary Outcomes

Secondary Outcomes (end points)
(1) Self-Esteem.
(2) Relationship with Friends.
(3) Expectations about the future at school.
(4) Summer activities.
Secondary Outcomes (explanation)
(1) This set of questions is intended to understand image the student has of themselves: their satisfaction with themselves, if they think they have good qualities, amongst other dimensions. In particular, they aim to give a glimpse of their socio-emotional status in what regards their self-image. Assessed through the agreement with a set of statements in a 5-Point Likert Scale (1 = Totally disagree; 2 = Disagree; 3 = Neither agree nor disagree; 4 = Agree; 5 = Totally agree).
(2) This set of questions is intended to understand the sense of belonging and general opinion of the pupil towards their friends/support system at school. In particular, they aim to show whether students are satisfied with the number, type and quality of social interactions they have with their peers. Assessed through the agreement with a set of statements in a 5-Point Likert Scale (1 = Totally disagree; 2 = Disagree; 3 = Neither agree nor disagree; 4 = Agree; 5 = Totally agree).
(3) This set of questions is intended to understand the students' expectations about the next school year, namely concerning their academic results and their social interactions. Assessed through the agreement with a set of statements in a 5-Point Likert Scale (1 = Totally disagree; 2 = Disagree; 3 = Neither agree nor disagree; 4 = Agree; 5 = Totally agree).
(4) This set of questions is intended to understand the students' past experiences with summer activities, as well as that of their siblings or peers. Moreover, we assess their notion of whether they believe their siblings or peers will take part in a summer school program during the following summer break. Mainly assessed through the agreement with a set of statements in a 5-Point Likert Scale (1 = Totally disagree; 2 = Disagree; 3 = Neither agree nor disagree; 4 = Agree; 5 = Totally agree), with one instance being reported through a 3-Point Likert Scale (1 = No; 2 = Don't know; 3 = Yes).

References:
(1) Rosenberg, Morris (1989). Society and the Adolescent Self-Image. Wesleyan University Press. ISBN: 978-0819562289.
(2) Marôco, João P. et al. (2014). “Adaptação Transcultural Brasil-Portugal da Escala de Satisfação com o Suporte Social para Estudantes do Ensino Superior”. In: Psicologia: Reflexão e Crítica 27 (2), pp. 247–256. ISSN: 0102-7972. DOI: 10.1590/1678-7153.201427205.

Experimental Design

Experimental Design
Our randomized controlled trial (RCT) focuses on Flemish primary school students (i.e. ages between 6 and 12 years old), with part of the analysis diving deeper into the sub-group of students coming from more disadvantaged backgrounds. In particular, we focus on primary school students due to the existing evidence that both COVID-19 and summer-related learning losses have disproportionately impacted younger students. The experiment will be carried out through a pre- and post-test strategy. The survey has been made in such a way that it is adapted for the younger audience within our population of interest, meaning that older subjects should not have any problem responding to it. The use of the same questionnaire is important for comparison purposes among individuals of different school grades.

In the last few weeks of the school year 2023/24, we have surveyed participants, gathering information on the students' demographic characteristics, mathematical cognitive ability, linguistic cognitive ability, socio-emotional state, previous partaking in summer schools or similar programs, opinion about their school, teachers and classmates, as well as expectations for the next academic year. The survey was divided into three parts. During the first part, students responded for the first time to the aforementioned set of questions. During the second part, students were exposed to a small prompt with images that varied according to their assignment to a control or treatment group, with treatment being randomly assigned at the student level using the randomization tool incorporated in the survey platform. Finally, students were asked a reduced number of questions intended to assess their attention during the treatment moment, followed by a repetition of a portion of the questions shown before the informational prompts, which was our way of assessing the effect of our treatment. Furthermore, questions aiming to grasp the individual's cognitive ability were conditional on the grade they were attending at school, at the moment of data collection.

Students were randomly assigned into five groups: control; treatment 1, which looked at the role of students' anchors tied to their opinions about their school; treatment 2, which was focused on the effect of our nudging strategy; treatment 3, which put together treatments 1 and 2; and treatment 4, which showed students how school can help them improve their socio-emotional state. The control group was exposed to no informational prompt at all. Those in treatment 1 were exposed to information on how school can help students learn better and how schools can be beneficial to them. Students assigned to treatment group 1 were exposed to a set of statements that shed a positive light on school as well, in an attempt to contradict this possible negative anchor. Those in treatment 2 were exposed to information about what are summer schools, what kinds of activities take place in them, the benefits of participating in summer schools, as well as some examples of alternative ways of spending their summer. Those in treatment 3 were exposed to all the information in both treatments 1 and 2. Finally, those in treatment group 4 were shown a series of socio-emotional statements tied to an explanation of how schools and summer schools can help develop those skills. Additionally, all students responded to a couple of questions intended to test their propensity for anchoring bias in the traditional sense (based on definition by Tversky and Kahneman, 1974), followed by a question on their opinion about their school, the treatment, and only then an inquiry concerning their likelihood of participation in a summer school program, which should allow us to capture the immediate effect of treatment 1. The test concerning their propensity for anchoring bias was designed such that students were shown a list of 15 simple pairs or words, they answered a couple of other questions (e.g. about their previous experiences over the summer), they were exposed to the question assigning them to a treatment group, and they were asked to state how many pairs of words they thought they could remember, in this order. Administrative data and data on the media campaign for summer schools served as supporting materials for our main research purpose as described above.
Experimental Design Details
Randomization Method
Treatment for both experiments were based on the randomizing capabilities of the survey platform (SurveyMonkey). For the Anchoring Experiment, this was done through a "Single Textbox A/B Test" type of question, with 1/3 probability assigned to each question. For the Summer School Experiment, this was done through the Page Randomization tool.
Randomization Unit
Individual.
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
700 pupils.
Sample size: planned number of observations
700 pupils.
Sample size (or number of clusters) by treatment arms
For the Anchoring Bias experiment: 233 pupils in the control group, 233 pupils in the low-anchor group, 234 pupils in the high-anchor group.
For the Summer School experiment: 140 pupils in the control group, 140 pupils in treatment group 1, 140 pupils in treatment group 2, 140 pupils in treatment group 3, 140 pupils in treatment group 4.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Power calculations made using the program G*Power. The following values correspond to calculations for: - Test Family - t tests - Statistical Test - Means: Difference between two independent means (two groups) - Type of Power Analysis - Sensitivity: Compute required effect size - given alpha, power, and sample size - Tail(s) - Two - Alpha err prob - 0.1 - Power (1 - beta err prob) - 0.8 - Sample size group 1: - Anchoring Experiment - 233 - Summer School Experiment - 140 - Sample size group 2: - Anchoring Experiment - 233 (for low-anchor group) and 234 (for high-anchor group) - Summer School Experiment - 140 For the Anchoring Bias experiment: 0.2306980 (for low-anchor group) and 0.2304507 (for high-anchor group) For the Summer School experiment: 0.2979087 References: (1) Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175-191. (2) Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009). Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41, 1149-1160.
Supporting Documents and Materials

Documents

Document Name
Survey
Document Type
survey_instrument
Document Description
Printed version of the survey. It includes a clear picture of the instruments used for treatment delivery. The version sees by the respondents was in color.
File
Survey

MD5: f96fef26f867d3bf858f332d849cbb44

SHA1: 437523b6a6981078c1d4777d8f4fd9203cb87bdd

Uploaded At: September 10, 2024

Document Name
GDPR compliance and ethical approval
Document Type
irb_protocol
Document Description
GDPR compliance and ethical approval, dated 2024-07-10.
File
GDPR compliance and ethical approval

MD5: c1f37cebc454d1b81910e1b2b4786b55

SHA1: f6c1fbe89ea0a211aded7ce310e0b678b871cb16

Uploaded At: September 10, 2024

IRB

Institutional Review Boards (IRBs)

IRB Name
Social and Societal Ethics Committee (SMEC) of KU Leuven
IRB Approval Date
2024-10-07
IRB Approval Number
G-2024-8035-R2(MAR)
Analysis Plan

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
Yes
Intervention Completion Date
June 29, 2024, 12:00 +00:00
Data Collection Complete
Yes
Data Collection Completion Date
June 29, 2024, 12:00 +00:00
Final Sample Size: Number of Clusters (Unit of Randomization)
For the Anchoring Bias experiment: 660 pupils.
For the Summer School experiment: 619 pupils.
Was attrition correlated with treatment status?
No
Final Sample Size: Total Number of Observations
For the Anchoring Bias experiment: 660 pupils.
For the Summer School experiment: 619 pupils.
Final Sample Size (or Number of Clusters) by Treatment Arms
For the Anchoring Bias experiment: 220 pupils in the control group, 222 pupils in the low-anchor group, 218 pupils in the high-anchor group. For the Summer School experiment: 133 pupils in the control group, 113 pupils in treatment group 1, 117 pupils in treatment group 2, 128 pupils in treatment group 3, 128 pupils in treatment group 4.
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
No
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials