The Case of College Coaching in the United States

Researchers partnered with twenty of the largest high schools in New Hampshire to examine whether in-person mentoring or mailings influence college enrollment decisions. 2,624 high school seniors were identified by guidance counselors at each school as having expressed an interest in attending college while making little to no progress in applying. Depending on the year of the study, roughly half of the identified students in each school were randomly assigned to one of two treatment programs while the remaining half were assigned to the control group.

In the mentoring program, Dartmouth College undergraduates met weekly with groups of high school seniors to provide advice and assistance in applying to universities. Application and test fees were paid upfront by the program, and participating high school students were offered a $100 cash bonus for completing the application process. In the mailing program, students could opt to have their transcripts sent to colleges in New Hampshire. Roughly twenty-five percent of these students (based on the quality of their transcript) received a letter from one of four selective four-year colleges in the state encouraging them to apply. In addition, all students in the mailing program were sent a letter from the New Hampshire Community College System highlighting the benefits of college and providing a URL to enable the student to apply. Intervention details are below:

Mentoring/ College Coaching Intervention: The main intervention consists of three components: mentoring, paying application and College Board/ACT fees, and a $100 cash bonus for completing the process. The process also includes starting the FAFSA. The most noticeable component (and most costly to implement) is in person mentoring by a Dartmouth College student. There is a team of roughly twenty Dartmouth students each year and most of these students worked full time on the project during January, February and part of March. For each high school, researchers choose a specific time and day of week to visit that school and all of the treatment students in that school. Visits are typically 2-3 hours in length and researchers promise up front to keep returning each week until every student has met his or her goals for college applications. The Dartmouth mentors keep track of each high school student's tasks, progress and various login IDs and passwords. Essays are often outlined during the mentoring session and further progress is made on essays at home. The specific steps required to "complete" the program include completing college essays, completing and filing at least one application, requesting transcripts and recommendation letters, sending College Board or ACT scores where appropriate, and starting the student section of the FAFSA and requesting a PIN (personal identification number) for the FAFSA. If students need to take the SAT or ACT, researchers help them sign up and provide email and phone reminders before the testing date. They pay for all SAT and ACT fees including additional costs of sending scores to schools. SAT fees and application fees are paid in real time for the high school students using the project’s credit cards. The program is not limited to applications to four-year colleges. Many students file applications to both two- and four- year colleges while some (roughly one-third) file applications at two-year colleges only. Almost all of the mentored time is spent completing college applications (often via the Common App), discussing and outlining college essays, sending SAT scores, sending transcripts, requesting recommendation letters, and filing the FAFSA. Most students finish the application process within 3-4 weeks.

Transcript Only/ Letter of Encouragement Intervention: In 2013 and 2014, researchers introduced another intervention designed to test whether the students in the sampling frame would be induced to attend college if they received a personalized letter of encouragement from one or more college admissions offices. Students in the “transcript only” intervention are nominated by guidance counselors as part of the same sample that is randomized to pure control or to mentoring treatment arms. Like the mentoring intervention, students selected for the transcript-only intervention are notified of their selection through email, in person notification by guidance counselors, and a letter/ release form, which is mailed to parents. If a student in the transcript only intervention agrees to participate, several steps occur: 1) The student fills out an online survey, which asks him/her to denote which of the participating colleges and universities interest him/her; 2) The student signs a form, which releases his/her transcript to allow researchers to send it to the participating colleges; and, 3) Researchers send all transcripts to all colleges, but highlight for each admissions office those students that showed a particular interest in that institution. All students receive a letter from the Community College System, which highlights the financial and non-pecuniary benefits of attending college and provides the URL to enable the student to apply. The Community College admission offices follow up the letter with emails and school visits to encourage the transcript only students to file an application. Based on transcript data, some fraction (roughly twenty five percent) of participating transcript-only students are selected by one of the selective four year institutions (among UNH, Keene State, Plymouth State, and Southern New Hampshire University) for additional encouragement. Those institutions send each selected student a letter stating that the admissions office has reviewed his/her transcript, considers him/her to be a strong applicant, and strongly encourages an application. Furthermore, most of these additional letters from admissions offices mention the possibilities of financial aid and explain that there are additional financial aid funds available if the student should choose to apply.

2009-01-01

2015-05-31

Enrollment: enrolled in any college and enrolled in a four-year college
Persistence: enrolled in three or more semesters of college, enrolled in college in both the first 365 days following high school graduation and also the second 365 days following graduation

Researchers were not able to employ all four treatment arms simultaneously within each cohort largely due to funding constraints. The majority of the students were randomized between the mentoring treatment versus control (i.e., no intervention). In 2012 students were randomized between the mentoring treatment versus the cash bonus only treatment. In 2013, students were randomized between the informational/ transcript only treatment and control. In 2014, students were randomized between the mentoring treatment and the informational/transcript only treatment. Mentoring treatment, cash bonus only, and transcript only/ informational students are notified by multiple methods (in person, over email, and via letters) from their guidance counselor that they have been selected for a Dartmouth College program intended to help them complete college applications. Mentoring students are told that the program includes in person mentoring, having college applications and College Board (or ACT) fees paid, and a $100 cash bonus for completing the process. The mentoring students in 2014 were not offered a cash bonus but were given all other aspects of the program. Pure control (no intervention) students are not contacted prior to their graduation out of concern that they would change their behavior or become upset that they were randomized out of receiving mentoring and a cash bonus. The Clearinghouse data, College Board data, and other NH Data Warehouse items are available for all students in the treatment and control groups.

Guidance departments provided researchers with student names and unique student ID numbers. For the mentoring treatment group researchers have data on the number of visits and the name and gender of the assigned mentor. Third, for all students researchers collected post-program survey data on parent's education, applications filed, acceptances received, and intended plans after high school graduation. They also collected post-program survey data on intended occupation, the student's estimate of annual income in that occupation and their belief as to whether a college degree was needed to succeed in that occupation. The survey also included a host of personality questions designed to elicit self-esteem, work ethic, and ability to meet deadlines. Researchers asked a battery of questions about sources of help and advice on careers and college going. Fourth, researchers have data from the New Hampshire Department of Education's Data Warehouse. These data include student gender, free lunch status, year of graduation, race, 10th grade math, reading and science scores, high school, and the year that the student first shows up in New Hampshire public schools. They also have SAT taking status, SAT scores, and the SAT Questionnaire data, the Data Warehouse data not just for the experimental sample, but also for every student in New Hampshire in the 2009-2014 graduation cohorts. The Data Warehouse also provides National Student Clearinghouse data on each college enrollment experienced by a student in the 2009-2014 cohorts. Clearinghouse data detail the college attended, dates of enrollment, two year versus four-year college, and any degrees earned. The Clearinghouse data cover 95 percent or more of enrollments at accredited colleges and universities.

The SAT Questionnaire data are useful in that they were mostly gathered administratively prior to the experiment. These SAT survey questions include desired level of education, whether the student wants to attend college close to home, involvement in sports and extracurricular activities, and whether the student needs help in forming educational plans. Researchers’ own survey data were gathered 0-24 months after students graduated from high school. Researchers worried that a pre-survey of both groups would alert the control students that they had been nominated to receive cash bonuses, payment of application fees, and mentoring but that they were randomly assigned to the control condition, which might affect students’ behavior or create resentment from not being chosen. Instead, researchers engaged in a comprehensive effort to contact students by email and Facebook following their high school graduation. To maximize the response rate researchers offered a $75 gift card to any of Amazon, Starbucks, J-Crew, or iTunes. Even with numerous contacts per student, our survey response rate is roughly 25 percent. To account for potential non-response bias researchers used propensity score weighting to weight the data by the inverse probability of responding.

There were 2,624 students in the experiment, with 871 of those students in the mentoring treatment group (45 percent of assigned students participated in the study), 851 students in the transcript-only group (14 percent of assigned students participated), and 902 students in the control group.

Researchers use a regression equation where the outcome variable is whether or not a student enrolls in college following graduation and whether or not a study enrolled in a four-year college after the intervention; the equation includes dummies for treatment arm (each of the three treatment groups, while the omitted category is the no intervention control group), high school* cohort fixed effects, and demographic characteristics (e.g., gender, nonwhite, age, free and reduced lunch status, and in some specifications 10th grade test scores). Standard errors are corrected for clustering at the high school*cohort level which is the level at which the experiment is run. Researchers control for age by including a full set of birth year*cohort dummies. Researcher use both OLS and probit regressions. This equation provides an intention to treat (ITT) estimate as only about half of the invited mentoring treatment students participate. Researchers also calculate treatment-on-the-treated estimates by instrumenting for participation in each treatment arm with dummy variables for assignment to the various treatment groups.

Most of analyses focus on outcomes of "ever enrolled" during the sample period as opposed to having separate dummies for enrolled in the first year after college, enrolled in the second year, etc. Naturally "ever enrolled" rises slightly as a cohort ages and researchers control for this with the inclusion of cohort dummies. As a robustness check, researchers also run all of analyses with dummies for "ever enrolled in the first year" or "ever enrolled in the first two years."

Researchers also assess whether the mentoring treatment is particularly effective for subgroups of students. The equation captures the direct effect of a particular student characteristic (e.g. having a college educated mother or “struggles to meet deadlines”) on college going and an interaction between that characteristic and the mentoring treatment.

Researchers define two different variables to measure persistence, in addition to just enrollment. For the graduating cohorts of 2009-2012, they create a dummy for enrollment in three or more semesters of college and a dummy for having enrolled in college in both the first 365 days following high school graduation and also the second 365 days following graduation.

Additionally, researchers assess how and why some treatments work or don’t. They interact treatment status with student characteristics and student answers to survey questions, keeping enrollment in any college as the outcome variable. For example, in the SAT Questionnaire, students are asked whether they anticipate needing outside or additional help forming educational plans. They also assess whether the treatment provides a boost of encouragement to students with low self-esteem; they interact treatment status with measures of self-esteem including “I am a person of worth equal to others” and “I can change important things.” Researchers also assess the interaction between personality and the effectiveness of the treatment through Openness to Experience.

Finally, they assess how the program interacts with demographic sources of advantage (e.g., having a high-school educated mother), how treatment effects vary by high school, and whether the cash bonus alone could generate the treatment effect, as well as conduct a cost-benefit analysis.

Excel random number generator

Student

No

NA; treatment is not clustered

2,624 students

Control: 902 students; Mentoring Treatment: 871 students; Transcript-Only Group: 851 students

An online Appendix to "Why Do College Going Interventions Work?"

Carrell, Scott E., and Bruce I. Sacerdote. Online Appendix to "Why Do College Going Interventions Work?" Working Paper, June 2016.

https://www.aeaweb.org/articles?id=10.1257/app.20150530

Researchers present evidence from a series of field experiments in college coaching/mentoring. They find large impacts on college attendance and persistence, but only in the treatments where they use an intensive boots on the ground approach to helping students. The treatments that provide financial incentives or information alone do not appear to be effective. For women, assignment to the mentoring treatment yields a 15 percentage point increase in the college going rate while treatment on the treated estimates are 30 percentage points (against a control complier mean rate of 43 percent). Researchers find much smaller treatment effects for men and the difference in treatment effects across genders is partially explained by the differential in self-reported labor market opportunities. Researchers do not find evidence that the treatment effect derives from simple behavioral mistakes, student disorganization, or a lack of easily obtained information. Instead the mentoring program appears to substitute for the potentially expensive and often missing ingredient of skilled parental or teacher time and encouragement.

Carrell, Scott E., and Bruce I. Sacerdote. "Why Do College Going Interventions Work?" Working Paper, June 2016.

https://www.aeaweb.org/articles?id=10.1257/app.20150530