Strengthening Students’ Resilience: Supporter text messages at Western Sydney University

Using text messages to motivate, inform, and remind individuals at key moments has shown to be a promising research area. BIT’s previous research in the area of education has demonstrated how a nudge to someone in a students’ support network can help improve educational outcomes for the student. In one study, 1,500 students in the UK were asked to nominate a “study supporter” - somebody whom they knew and who cared about their learning, such as a parent, grandparent, sibling or other family member, or friend.

These ‘Study Supporters’ were then texted weekly prompts to encourage the student who nominated them. The texts contained planning tips, upcoming deadlines, course content, academic resources and exam dates. These messages were designed to leverage existing social support networks that the students had but which were not, at the time, being put to good use. At the end of the school year, students who received the “study supporter” intervention were 27% more likely to pass their exams and 11% more likely to attend class.

This current trial is being conducted by the Behavioural Insights Team (BIT) as part of the Department of Social Service’s (DSS) Try Test Learn (TTL) Fund. DSS established the TTL fund in 2016. The purpose of the fund is to identify and help Australians who are at risk of long term welfare dependency, under the Australian Priority Investment Approach to Welfare. One of the groups identified were young students from disadvantaged backgrounds who are at risk of dropping out of their post-secondary studies.

This study aims to test whether personalised text messages to students and their nominated study supporters improves academic engagement and outcomes among Western Sydney University students, especially students from disadvantaged backgrounds.

External Link(s)

Registration Citation

Citation

Bradbury, Natalie et al. 2019. "Strengthening Students’ Resilience: Supporter text messages at Western Sydney University." AEA RCT Registry. July 19. https://doi.org/10.1257/rct.4467-1.0

Former Citation

Bradbury, Natalie et al. 2019. "Strengthening Students’ Resilience: Supporter text messages at Western Sydney University." AEA RCT Registry. July 19. https://www.socialscienceregistry.org/trials/4467/history/50423

Sponsors & Partners

Experimental Details

Interventions

Intervention(s)

This trial contains two trial arms: the treatment group (who nominate a Supporter and receive the text message intervention), and the control (who receive no intervention text messages).

The trial will run for a 17-week period during Spring Session 2019 of the WSU student calendar.

All students who opt in to participating in the study will complete a sign-up survey, and students will be stratified based on their responses to this survey. After being stratified, students will be randomly allocated into one of the following arms:

Control: students do not receive text messages; or
Treatment: students nominate a Supporter, and both the Supporter and the student receive text messages messages.

Intervention Start Date

2019-07-22

Intervention End Date

2020-07-31

Primary Outcomes

Primary Outcomes (end points)

There are two primary outcomes:

1. Do Supporter text messages improve unit achievement for those who received a grade? This takes the form of a final unit mark (0-100), which will be obtained through Western Sydney University administrative data at the end of Semester 2, 2019

2. Do Study Supporter text messages improve academic engagement? This takes the form of a binary outcome denoting whether the student completes the unit (yes/no). This will be obtained through Western Sydney University administrative data at the end of Semester 2, 2019

Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)

The secondary outcomes are listed below.

1. Do Study Supporter text messages improve grades in the other units a student takes. This will be measured using the average score in units not discussed in study supporter messages (0-100). This will be obtained by WSU administrative data at the end of Semester 2, 2019.

2. Do Study Supporter text messages improve academic engagement in the other units a student takes. This will be measured by the average completion rate in units not discussed in study supporter messages (0-1). This will be obtained using WSU administrative data at the end of Semester 2, 2019

3a. Do Study Supporter text messages increase student satisfaction with their academic performance. This is measured through self-reported satisfaction (1-7). The data will be obtained through a BIT survey of WSU students at the end of Semester 2, 2019.

3b. Do Study Supporter text messages increase student satisfaction with their academic performance. This is measured through self-reported impact of the intervention on course difficulty (1-7). The data will be obtained through a BIT survey of WSU students at the end of Semester 2, 2019.

4. Do study supporter text messages increase the students’ feelings of closeness with their nominated Study Supporter
Feelings of closeness (1-7)
BIT survey of WSU students at the end of Semester 2, 2019

Secondary Outcomes (explanation)

Experimental Design

The intervention will be evaluated with a two-armed randomised controlled trial (RCT), with randomisation at the student level and stratification by:
School
Gender
Age-group

Whether the student regularly talks to someone in their life about their studies.

Experimental Design Details

Randomization Method

After reading the Participant Information statement, and consenting to participate in the trial, students will fill out a brief survey in Qualtrics which captures the allocation variables of age (whether the student is 27 years of age or older), gender, school of the selected unit, and whether someone in the student’s life regularly asks about their studies.

Qualtrics will be programmed to stream students into 60 different strata based on these variables. Once students have been assigned to their strata, Qualtrics will randomise them to the treatment or control condition, using the inbuilt ‘randomiser’. Qualtrics will be set to distribute students within each strata evenly between treatment and control conditions. Students allocated to the control condition will be directed to the end of the survey, and notified that they are in the control group. Treatment students will be immediately directed to a second survey, where they will nominate a supporter, and provide their contact details. After completing this survey, treatment students will be directed to the end of the survey.

Randomization Unit

The allocation will be an individual level.

Was the treatment clustered?

Experiment Characteristics

Sample size: planned number of clusters

This trial is not clustered.

Sample size: planned number of observations

Based on the number of schools we are enrolling in the trial, we expect roughly 2,000 students to enroll in the trial, with 1,000 in each arm.

Sample size (or number of clusters) by treatment arms

Based on the number of schools we are enrolling in the trial, we expect roughly 2,000 students to enroll in the trial, with 1,000 in each arm.

Minimum detectable effect size for main outcomes (accounting for sample design and clustering)

We conducted power calculations to describe the relationship between the effect size, sample size, significance level and statistical power. This helped us to understand how many interventions we can test, and the minimum difference in the outcome measure that we can expect to be able to identify between arms. We have described only the power calculation for our primary outcome measure - a student’s grade on a scale of 0-100. Below we set out some assumptions which shape the calculations: Significance level: 0.05 Power: 0.80 Number of participants: Baseline rate: Average mark of 61.7 and average completion rate of 92% Number of treatment arms: 1 If we conservatively estimate a total sample size of 1,000 students (meaning an uptake rate of 14% by eligible students), we will be powered to detect an approximate 4 point increase in marks (e.g. increasing a student’s score from 65 to 69). If the sample size is 1,500 students (an uptake rate of 20%), we will be powered to detect a 3 point increase in mark and we will be powered to detect an approximate 1.5 percentage point increase in completion rates with 1,000 students, and a 1 percentage point increase with 1,500 students. We believe we are powered to detect realistic effect sizes. For example, BIT’s previous Study Supporter trial in the UK caused a six percentage point increase in students’ achievement rates, and a 3.1 percentage point increase in attendance rates.

Supporting Documents and Materials

IRB