Digital Messaging to Improve College Enrollment and Success

Last registered on September 25, 2020

Pre-Trial

Trial Information

General Information

Title
Digital Messaging to Improve College Enrollment and Success
RCT ID
AEARCTR-0006525
Initial registration date
September 25, 2020

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
September 25, 2020, 1:58 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Region

Primary Investigator

Affiliation
University of Pittsburgh

Other Primary Investigator(s)

PI Affiliation
University of Virginia
PI Affiliation
Harvard Graduate School of Education
PI Affiliation
Harvard Kennedy School

Additional Trial Information

Status
Completed
Start date
2014-09-01
End date
2020-09-01
Secondary IDs
Abstract
Setting: This study will include students from all across the United States, from a broad range of demographic backgrounds and urban and rural geographic locations.

Sample: Study participants include 75,000 students in the national sample, and 16,000 students in the Texas sample. The national sample will be representative of PSAT takers in lower-income schools who will be high school juniors in 2014-15. The Texas sample will include students from urban and rural districts serving large proportions of low-income students.

Intervention: Two text-messaging interventions, with one providing additional planning guidance for completing complex college-related activities, will be implemented in this study. In the national sample, equal numbers of students will receive each of the interventions, while students in the Texas sample will receive only the enhanced intervention. Beginning as high school juniors, students in Treatment Group 1 will receive text messages that provide reminders to complete discrete tasks related to college and financial aid (e.g., registering for college entrance exams). Students in Treatment Group 2 (enhanced) will receive these discrete task reminders, and a set of messages designed to encourage follow-through with more complex activities. Across both treatment groups, each text message will include the option to request help from a counselor, either at a call center (the national sample) or school (the Texas sample). Messages will continue through the summer after senior year. The text messages will include reminders and links to register for college entrance exams, apply for financial aid, and complete pre-matriculation tasks (e.g. orientation registration).

Research Design and Methods: The research team will implement a randomized controlled trial (RCT) design in order to estimate causal impacts of receiving a text-messaging intervention on students’ college-going outcomes. Researchers will randomize schools to Treatment Group 1, Treatment Group 2, or the Control Group in the national sample, and Treatment Group 2 or the Control Group in the Texas sample. After pairing schools, in groups of two or three, based on the college enrollment percentage in the high school class of 2013, they will randomly choose two schools from each pair within the national sample, and one school from each pair within the Texas sample to receive the intervention(s). Students will receive college-search and college-going advice as a function of the group to which their school was assigned. The research design will facilitate answering the following research questions:

What impact do text messages that give students information related to college search, application, FAFSA completion, and summer-specific college transition tasks have on whether high school students enroll in college, and which colleges students choose to attend?
Does the impact of text messaging on college-going outcomes differ by student characteristics?
How do students perceive the text messages? Do they find them helpful, and do they believe that the text messages impacted their postsecondary choices and preparation activities?
Are students who receive text messages more likely to seek assistance from a counselor than other students?
Control Condition: Students in the control group will receive the typical level of college-planning services offered by their school.

Key Measures: Investigators will examine the intervention's impact on both short-term process outcomes such as SAT registration, and longer-run outcomes such as college enrollment and persistence. The College Board will provide data on SAT registration and test-taking outcomes, and the National Student Clearinghouse will provide enrollment and persistence data. Researchers will also connect student responses to the text messages with data on completion of college-going tasks, to assess the mechanisms through which messaging influences students’ decisions and actions. Qualitative data will include responses to text-message queries, transcripts from student and counselor focus group panels, and responses to a post-intervention internet-based cell phone survey.

Data Analytic Strategy: With the data resulting from the RCT design, researchers will use regression analysis to assess intervention impacts. A multi-level model of students within schools will include students from the national and Texas samples, and will include student-level covariates to increase the efficiency of estimation. In addition, researchers will use qualitative methods to examine students’ text responses and perceptions of how the text messages influenced their college decisions. Qualitative researchers will code text message replies both inductively and deductively, by searching for emerging themes in student responses and by labeling responses according to hypothesized categories of student message use. They will code transcribed student and counselor focus group texts inductively. Multiple coders will be used for all qualitative data coding, and researchers will continue refining codes until ratings are reliable.
External Link(s)

Registration Citation

Citation
Avery, Christopher et al. 2020. "Digital Messaging to Improve College Enrollment and Success." AEA RCT Registry. September 25. https://doi.org/10.1257/rct.6525-1.0
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
Two text-messaging interventions, with one providing additional planning guidance for completing complex college-related activities, will be implemented in this study. In the national sample, equal numbers of students will receive each of the interventions, while students in the Texas sample will receive only the enhanced intervention. Beginning as high school juniors, students in Treatment Group 1 will receive text messages that provide reminders to complete discrete tasks related to college and financial aid (e.g., registering for college entrance exams). Students in Treatment Group 2 (enhanced) will receive these discrete task reminders, and a set of messages designed to encourage follow-through with more complex activities. Across both treatment groups, each text message will include the option to request help from a counselor, either at a call center (the national sample) or school (the Texas sample). Messages will continue through the summer after senior year. The text messages will include reminders and links to register for college entrance exams, apply for financial aid, and complete pre-matriculation tasks (e.g. orientation registration).
Intervention (Hidden)
Intervention Start Date
2015-04-01
Intervention End Date
2016-09-30

Primary Outcomes

Primary Outcomes (end points)
Investigators will examine the intervention's impact on both short-term process outcomes such as SAT registration and where possible FAFSA filing and college-application behavior; and longer-run outcomes such as college enrollment and persistence. The College Board will provide data on SAT registration and test-taking outcomes, and the National Student Clearinghouse will provide enrollment and persistence data.
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
The research team will implement a randomized controlled trial (RCT) design in order to estimate causal impacts of receiving a text-messaging intervention on students’ college-going outcomes. Researchers will randomize schools to Treatment Group 1, Treatment Group 2, or the Control Group in the national sample, and Treatment Group 2 or the Control Group in the Texas sample. After pairing schools, in groups of two or three, based on the college enrollment percentage in the high school class of 2013, they will randomly choose two schools from each pair within the national sample, and one school from each pair within the Texas sample to receive the intervention(s). Students will receive college-search and college-going advice as a function of the group to which their school was assigned.
Experimental Design Details
Randomization Method
Randomization was conducted using Stata software.
Randomization Unit
Randomization was conducted at the school level. Schools were first organized into groups based on prior levels of college going and then randomized to treatment or control within these groups. A similar strategy was employed in both of the CRTs.
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
National study: 745 schools
Texas study: 72 schools
Sample size: planned number of observations
National study: 70,285 students Texas study: 21,001 students
Sample size (or number of clusters) by treatment arms
National study: 386 treatment; 359 control
Texas study: 29 treatment; 43 control
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
We will assess the main effect of the student-focused intervention by investigating differences in outcomes such as timely postsecondary enrollment in a four-year college or university. The variable on which we blocked schools to create groups, a lagged measure of four-year enrollment, serves to explain 95% of the variation in the lagged measure. To be conservative, in these calculations, we assume that this blocking variable serves to explain 60 percent of the variation in the key outcomes of interest. With this assumption, and considering a range of effect size variability (from no effect size variability to effect size variability of .10) and a range of intraclass correlation (from .10 to .20), we estimate that we will have sufficient power (power = 0.80) to detect differences on the order of .047 to .09 standard deviations. Based on the average lagged four-year enrollment rate of 30 percent, this is an effect on the order of 2 to 4 percentage points.
IRB

Institutional Review Boards (IRBs)

IRB Name
Harvard University Area Institutional Review Board
IRB Approval Date
2014-08-01
IRB Approval Number
IRB14-2468

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
Yes
Intervention Completion Date
September 30, 2016, 12:00 +00:00
Data Collection Complete
Yes
Data Collection Completion Date
June 01, 2020, 12:00 +00:00
Final Sample Size: Number of Clusters (Unit of Randomization)
National study: 745 schools
Texas study: 72 schools
Was attrition correlated with treatment status?
No
Final Sample Size: Total Number of Observations
National study: 70,285 students
Texas study: 21,001 students
Final Sample Size (or Number of Clusters) by Treatment Arms
National study: 386 treatment; 359 control Texas study: 29 treatment; 43 control
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
No
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials