x

We are happy to announce that all trial registrations will now be issued DOIs (digital object identifiers). For more information, see here.
Addressing Barriers to Student Success in Higher Education
Last registered on December 03, 2018

Pre-Trial

Trial Information
General Information
Title
Addressing Barriers to Student Success in Higher Education
RCT ID
AEARCTR-0002744
Initial registration date
September 14, 2018
Last updated
December 03, 2018 1:00 PM EST
Location(s)
Primary Investigator
Affiliation
Oregon State University
Other Primary Investigator(s)
PI Affiliation
Reed College
Additional Trial Information
Status
In development
Start date
2018-09-20
End date
2019-06-30
Secondary IDs
Abstract
More than two of every five students who enroll in college fail to graduate within 6 years. Prior research has identified ineffective study habits as a major barrier to success. Using insights from behavioral economics, this study will assess whether focusing student attention on improving study habits can change behavior and improve outcomes. We will conduct a randomized controlled trial at Oregon State University to assess multiple interventions designed to overcome barriers to college completion. The interventions include encouragement to receive academic coaching on study strategies, to attend academic tutoring, and to increase study effort. We will also randomize the medium, timing, frequency, and offer of a lottery-based incentive for each encouragement, allowing us to measure how message delivery alters student behavior. Our findings will provide evidence on how to increase use of these services and the causal effects of these services.
External Link(s)
Registration Citation
Citation
Pugatch, Todd and Nicholas Wilson. 2018. "Addressing Barriers to Student Success in Higher Education." AEA RCT Registry. December 03. https://doi.org/10.1257/rct.2744-2.0.
Former Citation
Pugatch, Todd and Nicholas Wilson. 2018. "Addressing Barriers to Student Success in Higher Education." AEA RCT Registry. December 03. https://www.socialscienceregistry.org/trials/2744/history/38248.
Experimental Details
Interventions
Intervention(s)
1. Improved study strategies: encouragement to visit the university Academic Success Center for academic coaching, or the Economics Tutoring Lab for tutoring in Economics courses.
2. Additional study effort: encouragement to complete extra practice problems.

Encouragement will occur via email and text messages, as follows:
1. None: Current standard for notifications [control condition].
2. Email: Emails encouraging use of support service, where number of emails sent will be randomized.
3. Text message: Text encouragement, where number of messages will be randomized.
4. Email with incentive: Students will receive an email indicating that they will be entered in a lottery to receive $250 credit at the campus dining halls and bookstore if they access the support service before a specified date.
5. Text message with incentive: Students will receive the incentive encouragement via text.

Frequency and timing of messages will vary randomly within the 10 weeks of each academic term, as follows:
1. Week 3
2. Week 6
3. Week 9
4. Weeks 3/6
5. Weeks 3/9
6. Weeks 6/9
7. Weeks 3/6/9
Intervention Start Date
2018-10-08
Intervention End Date
2019-06-15
Primary Outcomes
Primary Outcomes (end points)
Take-up of academic services:
--visits to Academic Success Center
--visits to Economics Tutoring Lab
--usage of and performance on extra practice problems
Primary Outcomes (explanation)
Secondary Outcomes
Secondary Outcomes (end points)
Student academic performance:
--course grades
--graduation
Secondary Outcomes (explanation)
Course grades will include courses that are subject of intervention (Economics Principles courses) and subsequent Economics courses.
Experimental Design
Experimental Design
We will encourage students to take specific actions to improve their performance in introductory economics courses. All students will be eligible for the full range of academic support services and receive the current standard for notifications, but subsets of students will be encouraged to access particular services. We will also randomize the medium (email or text), timing, frequency, and offer of a lottery-based incentive for each encouragement, allowing us to measure how message delivery alters student behavior. See the description of interventions for additional details.
Experimental Design Details
Our research method is the randomized controlled experiment. We will encourage students to take specific actions to improve their performance in introductory economics courses. All students will be eligible for the full range of academic support services and receive the current standard for notifications, but subsets of students will be encouraged to access particular services. We will measure how this encouragement influences support service take-up and learning outcomes. To assign students to groups, we will first randomly assign all study participants to control (25%) or treatment (75%). Within the treatment group, we then randomly assign the characteristics of each treatment: medium, message, incentive, frequency, and timing. • Medium: Within the subsample of treated students who provide a mobile phone number, we randomly assign the message medium as: 1. Email (50%) 2. Text message (50%) Within the subsample of treated students who do not provide a phone number, we assign all students to the email group. • Message: we assign an encouragement message to each treated student. The messages encourage different academic support services: 1. Academic coaching (33.3%) 2. Economics peer tutoring (33.3%) 3. Extra practice problems (33.3%) • Incentive: we assign an incentive or no incentive to each treated student: 1. Incentive (50%): The encouragement message indicates that the student will be entered in a lottery to receive $250 credit at the campus dining halls and bookstore if they access the support service before a specified date. 2. No incentive (50%) • Frequency and timing: we randomly assign the frequency and timing of encouragement messages to each treated student within the 10 weeks of the term: 1. Week 3 (14.3%) 2. Week 6 (14.3%) 3. Week 9 (14.3%) 4. Weeks 3/6 (14.3%) 5. Weeks 3/9 (14.3%) 6. Weeks 6/9 (14.3%) 7. Weeks 3/6/9 (14.3%) Note that each treatment characteristic remains constant for each treated student during the term, regardless of message frequency. For instance, if a student is assigned to receive 3 messages, they will all be for the same support service, same incentive treatment, and via the same medium. Within the subsample of treated students who do not provide a phone number, there are thus 42 possible combinations of treatment characteristics (3 messages x 2 incentives x 7 frequency/timing combinations). Within the subsample of treated students who provide a phone number, these 42 combinations exist for both media, leaving 84 possible combinations. To find the probability that a randomly selected member of each subsample is assigned a particular combination of treatment characteristics, one would multiply the probability of treatment assignment times the corresponding probabilities of each characteristic. For instance, the probability that a student who provides a phone number is assigned to the email/coaching/incentive/Week 9 cell is: Pr(treated)*Pr(email)*Pr(coaching)*Pr(incentive)*Pr(Week 9) = .75*.5*.333*.5*.143 = .009 The number of treatment combinations is unlikely to divide evenly among study participants, particularly as we stratify treatment within course sections, which typically enroll 175-250 students. The actual sample sizes assigned to each group will therefore not match the a priori treatment probabilities exactly.
Randomization Method
randomization done in office by computer using random number generator
Randomization Unit
individual student enrolled in Economics Principles courses
Was the treatment clustered?
No
Experiment Characteristics
Sample size: planned number of clusters
3,000 students enrolled in 12 sections of Economics Principles courses (ECON 201/202 at Oregon State University, Corvallis campus)
Sample size: planned number of observations
3,000 students
Sample size (or number of clusters) by treatment arms
750 students in control group.

Remaining students will be divided evenly between cells defined by encouragement intervention, communication medium, and frequency/timing.

The encouragement interventions are:
1. academic coaching
2. Economics peer tutoring
3. extra practice

The communication media are:
1. email
2. text message
3. email, with lottery incentive
4. text message, with lottery incentive

The frequency/timing combinations during the 10 weeks of the term are:
1. Week 3
2. Week 6
3. Week 9
4. Weeks 3/6
5. Weeks 3/9
6. Weeks 6/9
7. Weeks 3/6/9

Within the subsample of treated students who do not provide a phone number, there are thus 42 possible combinations of treatment characteristics (3 messages x 2 incentives x 7 frequency/timing combinations). Within the subsample of treated students who provide a phone number, these 42 combinations exist for both media, leaving 84 possible combinations. We will divide students as evenly as possible among these treatments, though actual cell sizes may vary due to the large number of treatment combinations.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Minimum detectable effects (MDEs) between any two study arms for our intermediate and main outcomes: • Visit academic coaching or peer tutoring: 3.1 percentage points (assumed baseline mean: 10%) • Complete at least 80% of extra practice problems: 4.1 percentage points (assumed baseline mean: 20%) • Grade of D, F, or Withdraw: 4.5 percentage points (assumed baseline mean: 20%) All power calculations assume a sample size of 3,000 students, a significance level (i.e. alpha) of 10%, power (i.e. beta) of 80%, and study arms of equal size. Specifically, we assume three treatment arms (corresponding to each encouragement message) and a control group of 750 students each. Baseline means are approximations from past sections of introductory economics courses. The MDEs represent comparisons of a single treatment arm versus the control group.
IRB
INSTITUTIONAL REVIEW BOARDS (IRBs)
IRB Name
Oregon State University
IRB Approval Date
2018-02-08
IRB Approval Number
8402
Analysis Plan

There are documents in this trial unavailable to the public. Use the button below to request access to this information.

Request Information
Post-Trial
Post Trial Information
Study Withdrawal
Intervention
Is the intervention completed?
No
Is data collection complete?
Data Publication
Data Publication
Is public data available?
No
Program Files
Program Files
Reports and Papers
Preliminary Reports
Relevant Papers