Using Online and Text Message Coaching to Improve Student Outcomes During the COVID-19 Pandemic

Last registered on January 08, 2021

Pre-Trial

Trial Information

General Information

Title
Using Online and Text Message Coaching to Improve Student Outcomes During the COVID-19 Pandemic
RCT ID
AEARCTR-0007000
Initial registration date
January 07, 2021

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
January 08, 2021, 11:40 AM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Region

Primary Investigator

Affiliation
York University

Other Primary Investigator(s)

PI Affiliation
University of Waterloo
PI Affiliation
University of Toronto

Additional Trial Information

Status
Completed
Start date
2020-05-11
End date
2020-09-01
Secondary IDs
Abstract
Many college students struggle with mental health concerns, such as anxiety or a lack of belonging, and are often unaware of on-campus support services. The recent, rapid shift from in-person to online learning caused by the COVID-19 pandemic potentially exacerbates these issues. Focus-group evidence in the spring of 2020 from over 500 students at a large Canadian university shows that many students experienced loneliness, lacked structure, and felt unmotivated to continue coursework online when COVID-19 lockdown measures began. Motivated by these findings, we then designed an online module and text-messaging platform to help students navigate these problems, experimentally evaluating the effects of the program on mental health, well-being, and GPA in a sample of over 1,600 students enrolled in courses at two large Canadian universities during the summer of 2020. In addition to course grades, outcomes include self-reported measures of overall physical and mental health.
External Link(s)

Registration Citation

Citation
Logel, Christine , Philip Oreopoulos and Uros Petronijevic. 2021. "Using Online and Text Message Coaching to Improve Student Outcomes During the COVID-19 Pandemic." AEA RCT Registry. January 08. https://doi.org/10.1257/rct.7000
Experimental Details

Interventions

Intervention(s)
The intervention takes place at two Canadian universities among all students taking first-year economics courses during the summer semester of 2020. After completing a short warm-up survey, students are randomized into either a control group that receives a few additional survey questions or a treatment group that is asked to read stories of recent student experiences in isolation and how they overcame challenges and regained a positive sense of control. At the end of the survey, treated students are invited to provide their cell phone number to receive weekly messages on study tips and mental health advice. All students are then surveyed again at the end of the semester in a follow-up survey to gather data on mental health and well-being.
Intervention Start Date
2020-05-11
Intervention End Date
2020-08-15

Primary Outcomes

Primary Outcomes (end points)
Mental health and well-being outcomes gathered from the follow-up survey at end of summer semester.
Primary Outcomes (explanation)
1. Satisfaction
• All things considered, how satisfied are you with your life as a whole these days?
• All things considered, how satisfied are you with your experience at university this term?
• How satisfied are you with this academic performance?

Answers to each question above are measured on a 1-7 scale. We take the mean of the standardized versions of the three variables above as our final satisfaction measure.

2. Feelings of control
• In the last 4 weeks, how often have you been upset because of things that were outside of your control?
• In the last 4 weeks, how often have you felt that you were unable to control the important things in your life?

Answers to each question above are measured on a 1-5 scale. We take the mean of the standardized versions of the two variables above as our final control measure.

3. Efficacy in dealing with COVID
• The COVID-19 pandemic has, so far, made my life challenging
• My situation during the COVID-19 pandemic is better than most other students
• During the COVID-19 pandemic I am able to feel socially connected to friends and family
• During the COVID-19 pandemic I still feel part of the University of Toronto community
• My situation trying to adapt during COVID-19 has, so far, been challenging
• In the last 4 weeks, how do you feel your situation under COVID-19 has changed?

Answers to each question above are measured on a 1-5, 1-6, or 1-7 scale. We take the mean of the standardized versions of the six variables above as our final measure of success in dealing with covid-19 pandemic.

4. Stress & Anxiety
• How much stress would you say you have been experiencing in the last 4 weeks?
• How confident do you feel that you are able to cope with your stress on a day-to-day basis?
• Over the last 4 weeks, how often have you been bothered by the following problems? Feeling nervous, anxious, or on edge
• Over the last 4 weeks, how often have you been bothered by the following problems? Not being able to stop or control worrying
• Over the last 4 weeks, how often have you been bothered by the following problems? Little interest or pleasure in doing things
• Over the last 4 weeks, how often have you been bothered by the following problems? Feeling down, depressed, or hopeless

Answers to each question above are measured on a 1-4 or 1-7 scale. We take the mean of the standardized versions of the six variables above as our final measure of stress and anxiety.

5. Sense of Belonging
• I feel like I belong here at university
• The university wants me to be successful here
• I know where to go if I need academic advice right now
• I know where to go if I need personal advice right now
• The university does its best to help support me

Answers to each question above are measured on a 1-6 scale. We take the mean of the standardized versions of the five variables above as our final measure of belonging.

6. Social Connections
• In the last 4 weeks, how often do you feel disconnected from others?
• In the last 4 weeks how often do you feel that you lack companionship?

Answers to each question above are measured on a 1-4 scale. We take the mean of the standardized versions of the two variables above as our final measure of social connections.

7. Mental and Physical Health
• In the last 4 weeks, would you say your mental health is:
• In the last 4 weeks, would you say your physical health is:

Answers to each question above are measured on a 1-5 scale. We take the mean of the standardized versions of the two variables above as our final measure of health.

Secondary Outcomes

Secondary Outcomes (end points)
GPA and credits earned from summer semester
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
Students enrolled in 2020 summer semester courses at two large Canadian universities go online to complete a warmup exercise. After completing a short warm-up survey and declaring whether they give consent for their data to be used for anonymous research purposes, students are randomized into either a control group that receives a few additional survey questions or a treatment group that is asked to read stories of recent student experiences in isolation and how they overcame challenges and regained a positive sense of control. At the end of the survey, treated students are invited to provide their cell phone number to receive weekly messages of study tips and mental health advice. Students are then surveyed again (using the same online platform) at end of the course to gather measures of mental health and well-being, as well as academic outcomes.

All students in first-year economics courses during the summer semester of 2020 at the two universities are required to complete the online exercises in order to receive 2 percent of their course grade. The course grade is awarded only for completion of the exercises, regardless of whether students provide consent for the research team to use their data.

Subgroup analysis: Treatment effects on all outcome variables mentioned above will be explored in the main sample and across four subgroups:

- Students with high versus low initial locus of control.
- Students initially struggling with life during the COVID-19 pandemic.
- Students experiencing financial insecurity initially.
- International students.

Our focus-group evidence, gathered in the spring of 2020 (prior to the experiment) at University 1, showed that students who took control of their new situations during the pandemic lockdowns by establishing new routines and a sense of purpose coped much better overall. We therefore gathered baseline measures of students' Locus of Control with the aim of exploring heterogeneous treatment effects across students who feel more or less control over their lives. This is our primary subgroup of interest.

Secondary subgroup analysis involves the following groups:

- Students initially struggling with life during the COVID-19 pandemic.
- Students experiencing financial insecurity initially.
- International students.

We gathered data on these variables during the initial survey and will explore whether treatment was more effective among students initially struggling with life under the pandemic, experiencing financial insecurity during the pandemic, and among international students who potentially face different implications from the pandemic owing to travel restrictions and time-zone differences.
Experimental Design Details
Randomization Method
Randomization is done using random digits in student numbers.
Randomization Unit
Randomization is at the student level
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
none, although our sample is drawn from two different Canadian universities.
Sample size: planned number of observations
1700 students in total 900 at one university; 800 at other university
Sample size (or number of clusters) by treatment arms
The sample at each university is equally (and randomly) divided between treatment and control groups. The approximate breakdown is the following:

University 1, Control group students: 450
University 1, Treatment group students: 450
University 2, Control group students: 400
University 2, Treatment group students: 400
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
IRB

Institutional Review Boards (IRBs)

IRB Name
Office of Research Ethics, York University
IRB Approval Date
2020-04-28
IRB Approval Number
e2017 - 202
IRB Name
University of Toronto Social Sciences, Humanities, and Education Research Ethics Board
IRB Approval Date
2020-04-23
IRB Approval Number
30541

Post-Trial

Post Trial Information

Study Withdrawal

There are documents in this trial unavailable to the public. Use the button below to request access to this information.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials