Impact Evaluation of the Food Scholarship Program

Food for Change (FFC) is an initiative run by Houston Food Bank to help people at-risk of food insecurity find a path away from being at-risk of food insecurity and towards long-term stability. One of the programs operated through this initiative is the Food Scholarship (FS) program, where Houston Food Bank partners with colleges and universities to provide students with groceries to lower the financial stress of students’ grocery costs so that they are better equipped to complete their education and achieve financial independence. Specifically, the program aims to encourage continued enrollment in higher education and support academic success. FS distributes up to 30 pounds of groceries per week from FFC markets to students who may be at-risk of food insecurity. For this intervention, HFB will enact a new text message campaign that sends customized text messages to each participant. We are testing two types of text messages: proactive texts that aim to promote initial program engagement, and reactive texts that aim to promote program re-engagement after lapses in visits. These text message campaigns will be rolled out in the campuses of three college partners with the Houston Food Bank: The University of Houston, The University of Houston Downtown, and Houston City College.

2026-01-05

2026-06-15

Number of visits to Food Scholarship markets and other HFB agencies divided by the number of weeks of enrollment in the Food Scholarship program.

All students who enroll in the FS program for the first time between Jan 1, 2026 and May 4, 2026 will be randomized into one of our treatment arms. Because students enrolling earlier in the semester will have access to the food pantries for a longer period of time, our primary outcome is total visits / weeks enrolled. We will also analyze several other measures of visit frequency, including:
Total visits
Any visit
Any visit in the final month of the semester
Visits in the days immediately following the reactive text
Probability of visiting a market in the first, second, and third weeks following a text message

Academic outcomes such as GPA, enrollment status, and credit hours
Quality of life survey measures such a food security, financial security, mental health, and human flourishing

Note that any outcomes coming from the quality of life survey will only be observed for those completing the survey. In all such instances, we will use Heckman Selection models to account for potentially endogenous selection into survey completion. The random offering of a financial incentive to complete the survey will serve as an exclusion restriction.

Each of the three schools will conduct their typical outreach and marketing for this program. Students interested in the Food Scholarship program will complete an enrollment form, which includes reporting demographic information and completing a consent section. Upon successful submission of the enrollment form, the information will be sent instantaneously via an automatic workflow from an existing HFB spreadsheet into the roster spreadsheet for this study that will randomize each student into one of 8 treatment arms, with a secondary randomization in place for students assigned to receive reactive texts. We plan to compare outcomes between the treatment and control groups to assess the impact of the proactive and reactive text messages.

We will also send an endline survey to all enrolled students after completion of the semester. We will contact participants regarding the endline survey up to 5 times: two emails and three texts. Messaging will be identical for all participants, with the exception of the random subset offered payment. For this group, each message will contain an additional line about their compensatory offer. We are leveraging the random offering of compensation to avoid bias resulting from unequal survey response rates between treatment and control groups.

Not available

Randomization will occur immediately upon program enrollment, when the information recorded in the intake form is recorded in the roster sheet. This sheet will be pre-populated with a list of integers from 1 through 8, each corresponding to one of the eight treatment arms. The primary randomization list is generated in blocks of 72 assignments with a fixed, pre-specified distribution: values 1 and 2 appear 12 times each, and values 3–8 appear 8 times each. Within every block, assignments are randomly ordered using a Fisher–Yates shuffle driven by a seed-based, deterministic pseudo-random number generator (Mulberry32), ensuring uniform randomization and full reproducibility. Independent seeds are generated and logged for the primary randomization and for message sequence randomization to ensure mathematical independence between procedures. This procedure is repeated independently for 160 blocks, producing 11,520 total assignment slots. For each slot, the assignment value, block ID, within-block order, and a unique run identifier are recorded. The resulting list is generated once at the start of the study and stored in the roster sheet. As students enroll, they are assigned to the treatment arm indicated by the integer in the row corresponding to their position in the roster. This design ensures each student has a 1 in 6 probability of being assigned to treatment arms 1 and 2, and a 1 in 9 probability of being assigned to treatment arms 3-8. We go into further detail regarding this design in the Pre-Analysis Plan.

In addition to randomizing text message and incentive receipt, we will also randomize the ordering of the 3 reactive text messages. There are 6 possible orderings of the three messages, so among the roughly 2044 participants set to be assigned to receive reactive messages, roughly 340 will be assigned to receive each ordering. The message sequence list is randomly generated separately in blocks of 30, each containing an equal number of values 1–6 (five of each). Values within each block are permuted using the same method described above, with a different independently-generated seed, ensuring statistical independence from the primary randomization. Message sequences are assigned conditionally: only participants randomized to receive reactive texts (random number ≥ 5) are assigned a message sequence from this pool. Participants not receiving reactive texts are assigned a value of 0 for message sequence. This conditional assignment ensures that the distribution of message sequences is balanced among those actually receiving a reactive text, rather than across the full sample. Students assigned to receive reactive texts will have an equal probability of being assigned to one of the six message sequences; we do not include weighting in this randomization procedure.

Individual

No

0

The sample will cover all students enrolled in The University of Houston, The University of Houston Downtown, and Houston City College, who enroll into the Food Scholarship program for the first time during our study period (1/1/2026 through 5/4/2026). Historic enrollment records provided by HFB show approximately 4,600 students enrolled into the program across the three locations during the Spring 2025 semester. Since we anticipate a similar number of students enrolling, our estimated sample size is 4,600.

There are 4 texting treatment arms: none, proactive, reactive, and both. All students enrolled in the program will be invited to complete an endline survey, and 1/2 will be randomly offered to receive incentives for survey completion and 1/2 will receive no such offer incentives. Thus, there are 8 treatment arms in total and the anticipated share of students in each treatment arm is as follows:

No proactive + no reactive + no incentive: 1/6
No proactive + no reactive + incentive: 1/6
Proactive + no reactive + no incentive: 1/9
Proactive + no reactive + incentive: 1/9
No proactive + reactive + no incentive: 1/9
No proactive + reactive + incentive: 1/9
Proactive + reactive + no incentive: 1/9
Proactive + reactive + incentive: 1/9

The total estimated sample size is 4,600 with 1533 receiving no proactive or reactive text, while 1022 will be in each of {P NR,NP R,P R}; thus, roughly 2044 will be assigned to receive a proactive text and 2044 assigned to receive a reactive text. Furthermore, roughly 2300 will be offered to receive an incentive to complete the endline survey, while the other 2300 will not.

Our power calculations use administrative visit data from Spring 2025 to construct control group means. Students made an average of 2.75 visits per semester (SD = 4.06). Taking into account the number of weeks enrolled, we see an average of .179 visits per weeks enrolled (SD = .245). The design assigns: Control (C): 1/3 of sample → 1,533 students Proactive only (T1): 2/9 of sample → 1,022 students Reactive only (T2): 2/9 of sample → 1,022 students Proactive + Reactive (T3): 2/9 of sample → 1,022 students Our initial model includes indicators for T1, T2, and T3, with 1,022 students contributing to each treatment–control comparison. If interaction effects are null, we drop T3 and pool T1 and T2, resulting in 2,044 students contributing to treatment–control comparisons. We therefore report power calculations for both scenarios. MDEs represent the minimum detectable difference in mean outcomes between each treatment arm and control. 1. Continuous Outcome: Number of Market Visits power twomeans 2.75, n1(1533) n2(1022) sd(4.06) alpha(0.05) power(0.8) MDE: 0.456 visits Interpretation: We can detect an increase from 2.75 visits (control mean) to 3.2 visits (treatment). This represents 16.6% of the control mean. power twomeans 2.75, n1(1533) n2(2044) sd(4.06) alpha(0.05) power(0.8) MDE: 0.384 visits Interpretation: We can detect an increase from 2.75 visits (control mean) to 3.1 visits (treatment). This represents 14% of the control mean. 2. Continuous Outcome: Number of Market Visits/Weeks Enrolled power twomeans .179, n1(1533) n2(1022) sd(.245) alpha(0.05) power(0.8) MDE: 0.028 visits/week enrolled Interpretation: We can detect an increase from .179 visits/weeks enrolled (control mean) to .207 visits/weeks enrolled (treatment). This represents 15.6% of the control mean. power twomeans .179, n1(1533) n2(2044) sd(.245) alpha(0.05) power(0.8) MDE: 0.023 visits/week enrolled Interpretation: We can detect an increase from .179 visits/weeks enrolled (control mean) to .202 visits/weeks enrolled (treatment). This represents 12.8% of the control mean.