The Case of College Coaching in the United States

Last registered on January 26, 2017

Pre-Trial

Trial Information

General Information

Title
The Case of College Coaching in the United States
RCT ID
AEARCTR-0001913
Initial registration date
January 26, 2017

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
January 26, 2017, 9:08 AM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Primary Investigator

Affiliation
Dartmouth College

Other Primary Investigator(s)

PI Affiliation
University of California, Davis

Additional Trial Information

Status
Completed
Start date
2008-09-01
End date
2016-06-15
Secondary IDs
Abstract
Researchers present evidence from a series of field experiments in college coaching/mentoring. They find large impacts on college attendance and persistence, but only in the treatments where they use an intensive boots on the ground approach to helping students. The treatments that provide financial incentives or information alone do not appear to be effective. For women, assignment to the mentoring treatment yields a 15 percentage point increase in the college going rate while treatment on the treated estimates are 30 percentage points (against a control complier mean rate of 43 percent). Researchers find much smaller treatment effects for men and the difference in treatment effects across genders is partially explained by the differential in self-reported labor market opportunities. Researchers do not find evidence that the treatment effect derives from simple behavioral mistakes, student disorganization, or a lack of easily obtained information. Instead the mentoring program appears to substitute for the potentially expensive and often missing ingredient of skilled parental or teacher time and encouragement.
External Link(s)

Registration Citation

Citation
Carrell, Scott and Bruce Sacerdote. 2017. "The Case of College Coaching in the United States." AEA RCT Registry. January 26. https://doi.org/10.1257/rct.1913-1.0
Former Citation
Carrell, Scott and Bruce Sacerdote. 2017. "The Case of College Coaching in the United States." AEA RCT Registry. January 26. https://www.socialscienceregistry.org/trials/1913/history/13496
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
Researchers partnered with twenty of the largest high schools in New Hampshire to examine whether in-person mentoring or mailings influence college enrollment decisions. 2,624 high school seniors were identified by guidance counselors at each school as having expressed an interest in attending college while making little to no progress in applying. Depending on the year of the study, roughly half of the identified students in each school were randomly assigned to one of two treatment programs while the remaining half were assigned to the control group.

In the mentoring program, Dartmouth College undergraduates met weekly with groups of high school seniors to provide advice and assistance in applying to universities. Application and test fees were paid upfront by the program, and participating high school students were offered a $100 cash bonus for completing the application process. In the mailing program, students could opt to have their transcripts sent to colleges in New Hampshire. Roughly twenty-five percent of these students (based on the quality of their transcript) received a letter from one of four selective four-year colleges in the state encouraging them to apply. In addition, all students in the mailing program were sent a letter from the New Hampshire Community College System highlighting the benefits of college and providing a URL to enable the student to apply. Intervention details are below:

Mentoring/ College Coaching Intervention: The main intervention consists of three components: mentoring, paying application and College Board/ACT fees, and a $100 cash bonus for completing the process. The process also includes starting the FAFSA. The most noticeable component (and most costly to implement) is in person mentoring by a Dartmouth College student. There is a team of roughly twenty Dartmouth students each year and most of these students worked full time on the project during January, February and part of March. For each high school, researchers choose a specific time and day of week to visit that school and all of the treatment students in that school. Visits are typically 2-3 hours in length and researchers promise up front to keep returning each week until every student has met his or her goals for college applications. The Dartmouth mentors keep track of each high school student's tasks, progress and various login IDs and passwords. Essays are often outlined during the mentoring session and further progress is made on essays at home. The specific steps required to "complete" the program include completing college essays, completing and filing at least one application, requesting transcripts and recommendation letters, sending College Board or ACT scores where appropriate, and starting the student section of the FAFSA and requesting a PIN (personal identification number) for the FAFSA. If students need to take the SAT or ACT, researchers help them sign up and provide email and phone reminders before the testing date. They pay for all SAT and ACT fees including additional costs of sending scores to schools. SAT fees and application fees are paid in real time for the high school students using the project’s credit cards. The program is not limited to applications to four-year colleges. Many students file applications to both two- and four- year colleges while some (roughly one-third) file applications at two-year colleges only. Almost all of the mentored time is spent completing college applications (often via the Common App), discussing and outlining college essays, sending SAT scores, sending transcripts, requesting recommendation letters, and filing the FAFSA. Most students finish the application process within 3-4 weeks.

Transcript Only/ Letter of Encouragement Intervention: In 2013 and 2014, researchers introduced another intervention designed to test whether the students in the sampling frame would be induced to attend college if they received a personalized letter of encouragement from one or more college admissions offices. Students in the “transcript only” intervention are nominated by guidance counselors as part of the same sample that is randomized to pure control or to mentoring treatment arms. Like the mentoring intervention, students selected for the transcript-only intervention are notified of their selection through email, in person notification by guidance counselors, and a letter/ release form, which is mailed to parents. If a student in the transcript only intervention agrees to participate, several steps occur: 1) The student fills out an online survey, which asks him/her to denote which of the participating colleges and universities interest him/her; 2) The student signs a form, which releases his/her transcript to allow researchers to send it to the participating colleges; and, 3) Researchers send all transcripts to all colleges, but highlight for each admissions office those students that showed a particular interest in that institution. All students receive a letter from the Community College System, which highlights the financial and non-pecuniary benefits of attending college and provides the URL to enable the student to apply. The Community College admission offices follow up the letter with emails and school visits to encourage the transcript only students to file an application. Based on transcript data, some fraction (roughly twenty five percent) of participating transcript-only students are selected by one of the selective four year institutions (among UNH, Keene State, Plymouth State, and Southern New Hampshire University) for additional encouragement. Those institutions send each selected student a letter stating that the admissions office has reviewed his/her transcript, considers him/her to be a strong applicant, and strongly encourages an application. Furthermore, most of these additional letters from admissions offices mention the possibilities of financial aid and explain that there are additional financial aid funds available if the student should choose to apply.
Intervention Start Date
2009-01-01
Intervention End Date
2015-05-31

Primary Outcomes

Primary Outcomes (end points)
Enrollment: enrolled in any college and enrolled in a four-year college
Persistence: enrolled in three or more semesters of college, enrolled in college in both the first 365 days following high school graduation and also the second 365 days following graduation
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
Researchers were not able to employ all four treatment arms simultaneously within each cohort largely due to funding constraints. The majority of the students were randomized between the mentoring treatment versus control (i.e., no intervention). In 2012 students were randomized between the mentoring treatment versus the cash bonus only treatment. In 2013, students were randomized between the informational/ transcript only treatment and control. In 2014, students were randomized between the mentoring treatment and the informational/transcript only treatment. Mentoring treatment, cash bonus only, and transcript only/ informational students are notified by multiple methods (in person, over email, and via letters) from their guidance counselor that they have been selected for a Dartmouth College program intended to help them complete college applications. Mentoring students are told that the program includes in person mentoring, having college applications and College Board (or ACT) fees paid, and a $100 cash bonus for completing the process. The mentoring students in 2014 were not offered a cash bonus but were given all other aspects of the program. Pure control (no intervention) students are not contacted prior to their graduation out of concern that they would change their behavior or become upset that they were randomized out of receiving mentoring and a cash bonus. The Clearinghouse data, College Board data, and other NH Data Warehouse items are available for all students in the treatment and control groups.

Guidance departments provided researchers with student names and unique student ID numbers. For the mentoring treatment group researchers have data on the number of visits and the name and gender of the assigned mentor. Third, for all students researchers collected post-program survey data on parent's education, applications filed, acceptances received, and intended plans after high school graduation. They also collected post-program survey data on intended occupation, the student's estimate of annual income in that occupation and their belief as to whether a college degree was needed to succeed in that occupation. The survey also included a host of personality questions designed to elicit self-esteem, work ethic, and ability to meet deadlines. Researchers asked a battery of questions about sources of help and advice on careers and college going. Fourth, researchers have data from the New Hampshire Department of Education's Data Warehouse. These data include student gender, free lunch status, year of graduation, race, 10th grade math, reading and science scores, high school, and the year that the student first shows up in New Hampshire public schools. They also have SAT taking status, SAT scores, and the SAT Questionnaire data, the Data Warehouse data not just for the experimental sample, but also for every student in New Hampshire in the 2009-2014 graduation cohorts. The Data Warehouse also provides National Student Clearinghouse data on each college enrollment experienced by a student in the 2009-2014 cohorts. Clearinghouse data detail the college attended, dates of enrollment, two year versus four-year college, and any degrees earned. The Clearinghouse data cover 95 percent or more of enrollments at accredited colleges and universities.

The SAT Questionnaire data are useful in that they were mostly gathered administratively prior to the experiment. These SAT survey questions include desired level of education, whether the student wants to attend college close to home, involvement in sports and extracurricular activities, and whether the student needs help in forming educational plans. Researchers’ own survey data were gathered 0-24 months after students graduated from high school. Researchers worried that a pre-survey of both groups would alert the control students that they had been nominated to receive cash bonuses, payment of application fees, and mentoring but that they were randomly assigned to the control condition, which might affect students’ behavior or create resentment from not being chosen. Instead, researchers engaged in a comprehensive effort to contact students by email and Facebook following their high school graduation. To maximize the response rate researchers offered a $75 gift card to any of Amazon, Starbucks, J-Crew, or iTunes. Even with numerous contacts per student, our survey response rate is roughly 25 percent. To account for potential non-response bias researchers used propensity score weighting to weight the data by the inverse probability of responding.

There were 2,624 students in the experiment, with 871 of those students in the mentoring treatment group (45 percent of assigned students participated in the study), 851 students in the transcript-only group (14 percent of assigned students participated), and 902 students in the control group.

Researchers use a regression equation where the outcome variable is whether or not a student enrolls in college following graduation and whether or not a study enrolled in a four-year college after the intervention; the equation includes dummies for treatment arm (each of the three treatment groups, while the omitted category is the no intervention control group), high school* cohort fixed effects, and demographic characteristics (e.g., gender, nonwhite, age, free and reduced lunch status, and in some specifications 10th grade test scores). Standard errors are corrected for clustering at the high school*cohort level which is the level at which the experiment is run. Researchers control for age by including a full set of birth year*cohort dummies. Researcher use both OLS and probit regressions. This equation provides an intention to treat (ITT) estimate as only about half of the invited mentoring treatment students participate. Researchers also calculate treatment-on-the-treated estimates by instrumenting for participation in each treatment arm with dummy variables for assignment to the various treatment groups.

Most of analyses focus on outcomes of "ever enrolled" during the sample period as opposed to having separate dummies for enrolled in the first year after college, enrolled in the second year, etc. Naturally "ever enrolled" rises slightly as a cohort ages and researchers control for this with the inclusion of cohort dummies. As a robustness check, researchers also run all of analyses with dummies for "ever enrolled in the first year" or "ever enrolled in the first two years."

Researchers also assess whether the mentoring treatment is particularly effective for subgroups of students. The equation captures the direct effect of a particular student characteristic (e.g. having a college educated mother or “struggles to meet deadlines”) on college going and an interaction between that characteristic and the mentoring treatment.

Researchers define two different variables to measure persistence, in addition to just enrollment. For the graduating cohorts of 2009-2012, they create a dummy for enrollment in three or more semesters of college and a dummy for having enrolled in college in both the first 365 days following high school graduation and also the second 365 days following graduation.

Additionally, researchers assess how and why some treatments work or don’t. They interact treatment status with student characteristics and student answers to survey questions, keeping enrollment in any college as the outcome variable. For example, in the SAT Questionnaire, students are asked whether they anticipate needing outside or additional help forming educational plans. They also assess whether the treatment provides a boost of encouragement to students with low self-esteem; they interact treatment status with measures of self-esteem including “I am a person of worth equal to others” and “I can change important things.” Researchers also assess the interaction between personality and the effectiveness of the treatment through Openness to Experience.

Finally, they assess how the program interacts with demographic sources of advantage (e.g., having a high-school educated mother), how treatment effects vary by high school, and whether the cash bonus alone could generate the treatment effect, as well as conduct a cost-benefit analysis.
Experimental Design Details
Randomization Method
Excel random number generator
Randomization Unit
Student
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
NA; treatment is not clustered
Sample size: planned number of observations
2,624 students
Sample size (or number of clusters) by treatment arms
Control: 902 students; Mentoring Treatment: 871 students; Transcript-Only Group: 851 students
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
IRB

Institutional Review Boards (IRBs)

IRB Name
NBER
IRB Approval Date
2008-02-01
IRB Approval Number
FWA #00003692 IRB Ref#10-050

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
Yes
Intervention Completion Date
May 31, 2015, 12:00 +00:00
Data Collection Complete
Yes
Data Collection Completion Date
May 31, 2015, 12:00 +00:00
Final Sample Size: Number of Clusters (Unit of Randomization)
NA; treatment was not clustered
Was attrition correlated with treatment status?
Final Sample Size: Total Number of Observations
2,624 students
Final Sample Size (or Number of Clusters) by Treatment Arms
Control: 902 students; Mentoring Treatment: 871 students; Transcript-Only Group: 851 students
Data Publication

Data Publication

Is public data available?
Yes

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Program Files

Program Files
Yes
Reports, Papers & Other Materials

Relevant Paper(s)

Abstract
An online Appendix to "Why Do College Going Interventions Work?"
Citation
Carrell, Scott E., and Bruce I. Sacerdote. Online Appendix to "Why Do College Going Interventions Work?" Working Paper, June 2016.
Abstract
Researchers present evidence from a series of field experiments in college coaching/mentoring. They find large impacts on college attendance and persistence, but only in the treatments where they use an intensive boots on the ground approach to helping students. The treatments that provide financial incentives or information alone do not appear to be effective. For women, assignment to the mentoring treatment yields a 15 percentage point increase in the college going rate while treatment on the treated estimates are 30 percentage points (against a control complier mean rate of 43 percent). Researchers find much smaller treatment effects for men and the difference in treatment effects across genders is partially explained by the differential in self-reported labor market opportunities. Researchers do not find evidence that the treatment effect derives from simple behavioral mistakes, student disorganization, or a lack of easily obtained information. Instead the mentoring program appears to substitute for the potentially expensive and often missing ingredient of skilled parental or teacher time and encouragement.
Citation
Carrell, Scott E., and Bruce I. Sacerdote. "Why Do College Going Interventions Work?" Working Paper, June 2016.

Reports & Other Materials