Using Online and Text Message Coaching to Improve Student Outcomes During the COVID-19 Pandemic

The intervention takes place at two Canadian universities among all students taking first-year economics courses during the summer semester of 2020. After completing a short warm-up survey, students are randomized into either a control group that receives a few additional survey questions or a treatment group that is asked to read stories of recent student experiences in isolation and how they overcame challenges and regained a positive sense of control. At the end of the survey, treated students are invited to provide their cell phone number to receive weekly messages on study tips and mental health advice. All students are then surveyed again at the end of the semester in a follow-up survey to gather data on mental health and well-being.

2020-05-11

2020-08-15

Mental health and well-being outcomes gathered from the follow-up survey at end of summer semester.

1. Satisfaction
• All things considered, how satisfied are you with your life as a whole these days?
• All things considered, how satisfied are you with your experience at university this term?
• How satisfied are you with this academic performance?

Answers to each question above are measured on a 1-7 scale. We take the mean of the standardized versions of the three variables above as our final satisfaction measure.

2. Feelings of control
• In the last 4 weeks, how often have you been upset because of things that were outside of your control?
• In the last 4 weeks, how often have you felt that you were unable to control the important things in your life?

Answers to each question above are measured on a 1-5 scale. We take the mean of the standardized versions of the two variables above as our final control measure.

3. Efficacy in dealing with COVID
• The COVID-19 pandemic has, so far, made my life challenging
• My situation during the COVID-19 pandemic is better than most other students
• During the COVID-19 pandemic I am able to feel socially connected to friends and family
• During the COVID-19 pandemic I still feel part of the University of Toronto community
• My situation trying to adapt during COVID-19 has, so far, been challenging
• In the last 4 weeks, how do you feel your situation under COVID-19 has changed?

Answers to each question above are measured on a 1-5, 1-6, or 1-7 scale. We take the mean of the standardized versions of the six variables above as our final measure of success in dealing with covid-19 pandemic.

4. Stress & Anxiety
• How much stress would you say you have been experiencing in the last 4 weeks?
• How confident do you feel that you are able to cope with your stress on a day-to-day basis?
• Over the last 4 weeks, how often have you been bothered by the following problems? Feeling nervous, anxious, or on edge
• Over the last 4 weeks, how often have you been bothered by the following problems? Not being able to stop or control worrying
• Over the last 4 weeks, how often have you been bothered by the following problems? Little interest or pleasure in doing things
• Over the last 4 weeks, how often have you been bothered by the following problems? Feeling down, depressed, or hopeless

Answers to each question above are measured on a 1-4 or 1-7 scale. We take the mean of the standardized versions of the six variables above as our final measure of stress and anxiety.

5. Sense of Belonging
• I feel like I belong here at university
• The university wants me to be successful here
• I know where to go if I need academic advice right now
• I know where to go if I need personal advice right now
• The university does its best to help support me

Answers to each question above are measured on a 1-6 scale. We take the mean of the standardized versions of the five variables above as our final measure of belonging.

6. Social Connections
• In the last 4 weeks, how often do you feel disconnected from others?
• In the last 4 weeks how often do you feel that you lack companionship?

Answers to each question above are measured on a 1-4 scale. We take the mean of the standardized versions of the two variables above as our final measure of social connections.

7. Mental and Physical Health
• In the last 4 weeks, would you say your mental health is:
• In the last 4 weeks, would you say your physical health is:

Answers to each question above are measured on a 1-5 scale. We take the mean of the standardized versions of the two variables above as our final measure of health.

GPA and credits earned from summer semester

Students enrolled in 2020 summer semester courses at two large Canadian universities go online to complete a warmup exercise. After completing a short warm-up survey and declaring whether they give consent for their data to be used for anonymous research purposes, students are randomized into either a control group that receives a few additional survey questions or a treatment group that is asked to read stories of recent student experiences in isolation and how they overcame challenges and regained a positive sense of control. At the end of the survey, treated students are invited to provide their cell phone number to receive weekly messages of study tips and mental health advice. Students are then surveyed again (using the same online platform) at end of the course to gather measures of mental health and well-being, as well as academic outcomes.

All students in first-year economics courses during the summer semester of 2020 at the two universities are required to complete the online exercises in order to receive 2 percent of their course grade. The course grade is awarded only for completion of the exercises, regardless of whether students provide consent for the research team to use their data.

Subgroup analysis: Treatment effects on all outcome variables mentioned above will be explored in the main sample and across four subgroups:

- Students with high versus low initial locus of control.
- Students initially struggling with life during the COVID-19 pandemic.
- Students experiencing financial insecurity initially.
- International students.

Our focus-group evidence, gathered in the spring of 2020 (prior to the experiment) at University 1, showed that students who took control of their new situations during the pandemic lockdowns by establishing new routines and a sense of purpose coped much better overall. We therefore gathered baseline measures of students' Locus of Control with the aim of exploring heterogeneous treatment effects across students who feel more or less control over their lives. This is our primary subgroup of interest.

Secondary subgroup analysis involves the following groups:

- Students initially struggling with life during the COVID-19 pandemic.
- Students experiencing financial insecurity initially.
- International students.

We gathered data on these variables during the initial survey and will explore whether treatment was more effective among students initially struggling with life under the pandemic, experiencing financial insecurity during the pandemic, and among international students who potentially face different implications from the pandemic owing to travel restrictions and time-zone differences.

Randomization is done using random digits in student numbers.

Randomization is at the student level

No

none, although our sample is drawn from two different Canadian universities.

1700 students in total 900 at one university; 800 at other university

The sample at each university is equally (and randomly) divided between treatment and control groups. The approximate breakdown is the following:

University 1, Control group students: 450
University 1, Treatment group students: 450
University 2, Control group students: 400
University 2, Treatment group students: 400