Experimental Design
Researchers were not able to employ all four treatment arms simultaneously within each cohort largely due to funding constraints. The majority of the students were randomized between the mentoring treatment versus control (i.e., no intervention). In 2012 students were randomized between the mentoring treatment versus the cash bonus only treatment. In 2013, students were randomized between the informational/ transcript only treatment and control. In 2014, students were randomized between the mentoring treatment and the informational/transcript only treatment. Mentoring treatment, cash bonus only, and transcript only/ informational students are notified by multiple methods (in person, over email, and via letters) from their guidance counselor that they have been selected for a Dartmouth College program intended to help them complete college applications. Mentoring students are told that the program includes in person mentoring, having college applications and College Board (or ACT) fees paid, and a $100 cash bonus for completing the process. The mentoring students in 2014 were not offered a cash bonus but were given all other aspects of the program. Pure control (no intervention) students are not contacted prior to their graduation out of concern that they would change their behavior or become upset that they were randomized out of receiving mentoring and a cash bonus. The Clearinghouse data, College Board data, and other NH Data Warehouse items are available for all students in the treatment and control groups.
Guidance departments provided researchers with student names and unique student ID numbers. For the mentoring treatment group researchers have data on the number of visits and the name and gender of the assigned mentor. Third, for all students researchers collected post-program survey data on parent's education, applications filed, acceptances received, and intended plans after high school graduation. They also collected post-program survey data on intended occupation, the student's estimate of annual income in that occupation and their belief as to whether a college degree was needed to succeed in that occupation. The survey also included a host of personality questions designed to elicit self-esteem, work ethic, and ability to meet deadlines. Researchers asked a battery of questions about sources of help and advice on careers and college going. Fourth, researchers have data from the New Hampshire Department of Education's Data Warehouse. These data include student gender, free lunch status, year of graduation, race, 10th grade math, reading and science scores, high school, and the year that the student first shows up in New Hampshire public schools. They also have SAT taking status, SAT scores, and the SAT Questionnaire data, the Data Warehouse data not just for the experimental sample, but also for every student in New Hampshire in the 2009-2014 graduation cohorts. The Data Warehouse also provides National Student Clearinghouse data on each college enrollment experienced by a student in the 2009-2014 cohorts. Clearinghouse data detail the college attended, dates of enrollment, two year versus four-year college, and any degrees earned. The Clearinghouse data cover 95 percent or more of enrollments at accredited colleges and universities.
The SAT Questionnaire data are useful in that they were mostly gathered administratively prior to the experiment. These SAT survey questions include desired level of education, whether the student wants to attend college close to home, involvement in sports and extracurricular activities, and whether the student needs help in forming educational plans. Researchers’ own survey data were gathered 0-24 months after students graduated from high school. Researchers worried that a pre-survey of both groups would alert the control students that they had been nominated to receive cash bonuses, payment of application fees, and mentoring but that they were randomly assigned to the control condition, which might affect students’ behavior or create resentment from not being chosen. Instead, researchers engaged in a comprehensive effort to contact students by email and Facebook following their high school graduation. To maximize the response rate researchers offered a $75 gift card to any of Amazon, Starbucks, J-Crew, or iTunes. Even with numerous contacts per student, our survey response rate is roughly 25 percent. To account for potential non-response bias researchers used propensity score weighting to weight the data by the inverse probability of responding.
There were 2,624 students in the experiment, with 871 of those students in the mentoring treatment group (45 percent of assigned students participated in the study), 851 students in the transcript-only group (14 percent of assigned students participated), and 902 students in the control group.
Researchers use a regression equation where the outcome variable is whether or not a student enrolls in college following graduation and whether or not a study enrolled in a four-year college after the intervention; the equation includes dummies for treatment arm (each of the three treatment groups, while the omitted category is the no intervention control group), high school* cohort fixed effects, and demographic characteristics (e.g., gender, nonwhite, age, free and reduced lunch status, and in some specifications 10th grade test scores). Standard errors are corrected for clustering at the high school*cohort level which is the level at which the experiment is run. Researchers control for age by including a full set of birth year*cohort dummies. Researcher use both OLS and probit regressions. This equation provides an intention to treat (ITT) estimate as only about half of the invited mentoring treatment students participate. Researchers also calculate treatment-on-the-treated estimates by instrumenting for participation in each treatment arm with dummy variables for assignment to the various treatment groups.
Most of analyses focus on outcomes of "ever enrolled" during the sample period as opposed to having separate dummies for enrolled in the first year after college, enrolled in the second year, etc. Naturally "ever enrolled" rises slightly as a cohort ages and researchers control for this with the inclusion of cohort dummies. As a robustness check, researchers also run all of analyses with dummies for "ever enrolled in the first year" or "ever enrolled in the first two years."
Researchers also assess whether the mentoring treatment is particularly effective for subgroups of students. The equation captures the direct effect of a particular student characteristic (e.g. having a college educated mother or “struggles to meet deadlines”) on college going and an interaction between that characteristic and the mentoring treatment.
Researchers define two different variables to measure persistence, in addition to just enrollment. For the graduating cohorts of 2009-2012, they create a dummy for enrollment in three or more semesters of college and a dummy for having enrolled in college in both the first 365 days following high school graduation and also the second 365 days following graduation.
Additionally, researchers assess how and why some treatments work or don’t. They interact treatment status with student characteristics and student answers to survey questions, keeping enrollment in any college as the outcome variable. For example, in the SAT Questionnaire, students are asked whether they anticipate needing outside or additional help forming educational plans. They also assess whether the treatment provides a boost of encouragement to students with low self-esteem; they interact treatment status with measures of self-esteem including “I am a person of worth equal to others” and “I can change important things.” Researchers also assess the interaction between personality and the effectiveness of the treatment through Openness to Experience.
Finally, they assess how the program interacts with demographic sources of advantage (e.g., having a high-school educated mother), how treatment effects vary by high school, and whether the cash bonus alone could generate the treatment effect, as well as conduct a cost-benefit analysis.