Experimental Design
The impact evaluation was conducted as a randomized controlled trial. The full sample was comprised of 801 Free and Reduced Lunch (FRL) eligible 8th grade students from SBCSC and PHM school districts who had not yet enrolled in 21st Century Scholars. On June 1, 2017, the school districts sent researchers the universe of FRL eligible 8th grade students minus students that the schools were able to confirm had already enrolled in 21C. This produced the full sample. Out of this sample, 266 were randomly assigned by lottery to the Control Group, 265 were assigned to the Letter Group, and 270 were assigned to the Call Group. Most schools within both school districts perform a certain amount of advertising for the 21st Century Scholars program in the absence of this intervention at the baseline. This baseline advertising for 21C usually occurs in the form of a letter mailed home to parents. These letters typically direct parents to the 21C website and reiterate the deadline to enroll (June 30 following the child’s eighth grade year), however, they do not include any personalized information specific to the student. These letters are mailed in the fall and the winter several months before the deadline.
The nudge intervention for college scholarship program designed a new form of contact that is distinct from baseline outreach in several intentional ways. Extensive research went into the drafting of the nudges. Guidance counselors from both PHM and SBCSC school districts provided context to the issue of low enrollment: their experience working with low-income families suggested that there was a serious information gap that prevents parents from enrolling their children in the program. First and foremost, guidance counselors reported that many families seemed to be unaware of the program or did not fully understand the benefit. Second, many eligible students would be first-generation college students, and their parents might underestimate the value of a college education. Additionally, in order to enroll, parents must know the nine-digit student test number (STN) of their child, their four-digit middle school code, and their four-digit high school code. None of these numbers are readily or easily available to parents, and require parents to dig through student test reports or search online for them. Finally, the guidance counselors also suggested that parents were prone to forget the deadline or lose the necessary application information long before the deadline. In general, it is harder to reach low SES families as schools are more likely to lack reliable contact information for this demographic. Low SES populations are notoriously mobile, increasing the likelihood that the school does not have an up-to-date address on file. Additionally, many low SES parents do not answer their telephones, or have listed numbers that have since been discontinued.
Both the letter nudge and the phone call nudge were designed to include four main components in an attempt to abate some of the previously stated obstacles: (1) a clear explanation of the program, including a dollar amount for the average benefit that the student is entitled to (in the case of this study, the average benefit was $35,000 in scholarship funds towards tuition to cover the cost of public school tuition in Indiana); (2) the research-supported difference in lifetime earnings for someone with a college degree compared to someone with only a high school degree (including the dollar value of $1 million); (3) personalized information to the student: the child’s personal nine-digit STN, the relevant four-digit middle school code, and the relevant four-digit high school code; and (4) the enrollment website and deadline reiterated. This information was laid out as such in the letters that were sent home to parents of students in the Letter Group, and the Call Group received a nearly verbatim verbal version of the letter over the phone.
In practice, the letters were sent out by the school on the Monday of the week that the phone calls were made, approximately two weeks prior to the June 30th enrollment deadline. The timing of the nudge was also strategically closer to the deadline in order to encourage urgency in enrollment. While there is no way to verify that letters reached their intended destinations, callers employed a protocol to increase the number of successful calls. Up to two calls were made to any given number in the Call Group. If there was no answer when the first call was made, callers were instructed not to leave a voicemail. The number was then added back into the pool of calls to be made and marked as needing to receive a second call. If there was still no answer on the second call, callers were instructed to then leave a voicemail using a short form of the phone script that still contained each of the four main components from the phone script. Because calls were made from the school district headquarters at specific times during the day, callers were unable to provide a callback number, and were not instructed to suggest that parents try to return the call. In some cases, voice mailboxes were full or had not been set up, so callers were unable to leave a voicemail message. Some numbers did not work at all, and appeared to have been disconnected. In those cases, callers were not able to make any contact at all with the parents. In instances where the person called did not speak English, the caller ended the call and referred the number to a Spanish-speaking caller who called back using a Spanish version of the phone script. The first Spanish call counted as the first call in these cases. Voice mailboxes set up in Spanish also received a Spanish version of the voicemail message if there was no answer for either of the two Spanish calls. There was not a Spanish version of the letter mailed. Since eligible students were randomized at the individual level and not at the family level, some parents who received phone calls asked for the STN of a twin, sibling, or resident cousin who had not been randomized into the Call Group, unlike the child on whose behalf they were receiving the call. In these cases, because callers did not have the relevant STN data for children who were not randomized into the Call Group, parents were encouraged to call the child’s school and request the STN. We recognize that there will therefore be some small spillover effects represented in the results.
Observations from the nudge outreach staff confirmed the prevalence of an informational gap at play. Many parents were unaware of the 21C program, others had heard of the program but lacked basic information needed to fill out the program application, and others still had heard of the program but forgotten about the approaching deadline. Anecdotally, the callers reported overwhelmingly positive reactions from parents with whom they spoke on the phone. Those who had never heard of the program before seemed hopeful that college would be an option for their children. Those who had struggled with completing the application were thankful to be provided with the information needed to finalize their applications.