Addressing Barriers to Student Success in Higher Education

1. Improved study strategies: encouragement to visit the university Academic Success Center for academic coaching, or the Economics Tutoring Lab for tutoring in Economics courses.
2. Additional study effort: encouragement to complete extra practice problems.

Encouragement will occur via email and text messages, as follows:
1. None: Current standard for notifications [control condition].
2. Email: Emails encouraging use of support service, where number of emails sent will be randomized.
3. Text message: Text encouragement, where number of messages will be randomized.
4. Email with incentive: Students will receive an email indicating that they will be entered in a lottery to receive $250 credit at the campus dining halls and bookstore if they access the support service before a specified date.
5. Text message with incentive: Students will receive the incentive encouragement via text.

Frequency and timing of messages will vary randomly within the 10 weeks of each academic term, as follows:
1. Week 3
2. Week 6
3. Week 9
4. Weeks 3/6
5. Weeks 3/9
6. Weeks 6/9
7. Weeks 3/6/9

2018-10-08

2019-06-15

Take-up of academic services:
--visits to Academic Success Center
--visits to Economics Tutoring Lab
--usage of and performance on extra practice problems

Student academic performance:
--course grades
--graduation

Course grades will include courses that are subject of intervention (Economics Principles courses) and subsequent Economics courses.

We will encourage students to take specific actions to improve their performance in introductory economics courses. All students will be eligible for the full range of academic support services and receive the current standard for notifications, but subsets of students will be encouraged to access particular services. We will also randomize the medium (email or text), timing, frequency, and offer of a lottery-based incentive for each encouragement, allowing us to measure how message delivery alters student behavior. See the description of interventions for additional details.

Our research method is the randomized controlled experiment. We will encourage students to take specific actions to improve their performance in introductory economics courses. All students will be eligible for the full range of academic support services and receive the current standard for notifications, but subsets of students will be encouraged to access particular services. We will measure how this encouragement influences support service take-up and learning outcomes.

To assign students to groups, we will first randomly assign all study participants to control (25%) or treatment (75%). Within the treatment group, we then randomly assign the characteristics of each treatment: medium, message, incentive, frequency, and timing.

• Medium: Within the subsample of treated students who provide a mobile phone number, we randomly assign the message medium as:
1. Email (50%)
2. Text message (50%)
Within the subsample of treated students who do not provide a phone number, we assign all students to the email group.

• Message: we assign an encouragement message to each treated student. The messages encourage different academic support services:
1. Academic coaching (33.3%)
2. Economics peer tutoring (33.3%)
3. Extra practice problems (33.3%)

• Incentive: we assign an incentive or no incentive to each treated student:
1. Incentive (50%): The encouragement message indicates that the student will be entered in a lottery to receive $250 credit at the campus dining halls and bookstore if they access the support service before a specified date.
2. No incentive (50%)

• Frequency and timing: we randomly assign the frequency and timing of encouragement messages to each treated student within the 10 weeks of the term:
1. Week 3 (14.3%)
2. Week 6 (14.3%)
3. Week 9 (14.3%)
4. Weeks 3/6 (14.3%)
5. Weeks 3/9 (14.3%)
6. Weeks 6/9 (14.3%)
7. Weeks 3/6/9 (14.3%)

Note that each treatment characteristic remains constant for each treated student during the term, regardless of message frequency. For instance, if a student is assigned to receive 3 messages, they will all be for the same support service, same incentive treatment, and via the same medium.

Within the subsample of treated students who do not provide a phone number, there are thus 42 possible combinations of treatment characteristics (3 messages x 2 incentives x 7 frequency/timing combinations). Within the subsample of treated students who provide a phone number, these 42 combinations exist for both media, leaving 84 possible combinations.

To find the probability that a randomly selected member of each subsample is assigned a particular combination of treatment characteristics, one would multiply the probability of treatment assignment times the corresponding probabilities of each characteristic. For instance, the probability that a student who provides a phone number is assigned to the email/coaching/incentive/Week 9 cell is:

Pr(treated)*Pr(email)*Pr(coaching)*Pr(incentive)*Pr(Week 9) = .75*.5*.333*.5*.143 = .009

The number of treatment combinations is unlikely to divide evenly among study participants, particularly as we stratify treatment within course sections, which typically enroll 175-250 students. The actual sample sizes assigned to each group will therefore not match the a priori treatment probabilities exactly.

randomization done in office by computer using random number generator

individual student enrolled in Economics Principles courses

No

3,000 students enrolled in 12 sections of Economics Principles courses (ECON 201/202 at Oregon State University, Corvallis campus)

3,000 students

750 students in control group.

Remaining students will be divided evenly between cells defined by encouragement intervention, communication medium, and frequency/timing.

The encouragement interventions are:
1. academic coaching
2. Economics peer tutoring
3. extra practice

The communication media are:
1. email
2. text message
3. email, with lottery incentive
4. text message, with lottery incentive

The frequency/timing combinations during the 10 weeks of the term are:
1. Week 3
2. Week 6
3. Week 9
4. Weeks 3/6
5. Weeks 3/9
6. Weeks 6/9
7. Weeks 3/6/9

Within the subsample of treated students who do not provide a phone number, there are thus 42 possible combinations of treatment characteristics (3 messages x 2 incentives x 7 frequency/timing combinations). Within the subsample of treated students who provide a phone number, these 42 combinations exist for both media, leaving 84 possible combinations. We will divide students as evenly as possible among these treatments, though actual cell sizes may vary due to the large number of treatment combinations.

Minimum detectable effects (MDEs) between any two study arms for our intermediate and main outcomes: • Visit academic coaching or peer tutoring: 3.1 percentage points (assumed baseline mean: 10%) • Complete at least 80% of extra practice problems: 4.1 percentage points (assumed baseline mean: 20%) • Grade of D, F, or Withdraw: 4.5 percentage points (assumed baseline mean: 20%) All power calculations assume a sample size of 3,000 students, a significance level (i.e. alpha) of 10%, power (i.e. beta) of 80%, and study arms of equal size. Specifically, we assume three treatment arms (corresponding to each encouragement message) and a control group of 750 students each. Baseline means are approximations from past sections of introductory economics courses. The MDEs represent comparisons of a single treatment arm versus the control group.

More than two of every five students who enroll in college fail to graduate within six years. Prior research has identified ineffective study habits as a major barrier to success. We conducted a randomized controlled advertising experiment designed to increase demand for academic support services among more than 2,100 students at a large U.S. public university. Our results reveal several striking findings. First, the intervention shifted proxies of student attention, such as opening emails and self-reported awareness of service availability.However, the experimental variation indicates that approximately one-third of students are never attentive to student services. Second, advertising increased the use of extra practice problems, but did not affect take-up of tutoring and coaching, the other two services. Structural estimates suggest that transaction costs well in excess of plausible opportunity costs explain the differences in service use. Third, the characteristics of advertising messages matter. Several common nudging techniques—such as text messages, lottery-based economic incentives, and repeated messages—either had no effect or in some cases reduced the effectiveness of messaging.

Pugatch, Todd and Wilson, Nicholas, Nudging Demand for Academic Support Services: Experimental and Structural Evidence from Higher Education. IZA Discussion Paper No. 13732, Available at SSRN: https://ssrn.com/abstract=3699849

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3699849

We conducted an experiment designed to increase demand for academic support services among more than 2,100 students at a large U.S. public university. The intervention shifted student attention and increased service use, but also revealed behavioral biases. Structural estimates using the experimental variation suggest that transaction costs well in excess of plausible opportunity costs explain relatively low service use. Moreover, one-third of students are never attentive to student services. Message characteristics also matter. Several common nudging techniques—such as text messages, lottery-based economic incentives, and repeated messages—either had no effect or in some cases reduced the effectiveness of messaging.

Todd Pugatch, Nicholas Wilson. "Nudging Demand for Academic Support Services: Experimental and Structural Evidence from Higher Education." Journal of Human Resources, Published online before print March 09, 2022.

https://doi.org/10.3368/jhr.0221-11474R2