Impact Evaluation of the Food Scholarship Program

Food for Change (FFC) is an initiative run by Houston Food Bank to help people at-risk of food insecurity find a path away from being at-risk of food insecurity and towards long-term stability. One of the programs operated through this initiative is the Food Scholarship (FS) program, where Houston Food Bank partners with colleges and universities to provide students with groceries to lower the financial stress of students’ grocery costs so that they are better equipped to complete their education and achieve financial independence. Specifically, the program aims to encourage continued enrollment in higher education and support academic success. FS distributes up to 30 pounds of groceries per week from FFC markets to students who may be at-risk of food insecurity. For this intervention, HFB will enact a new text message campaign that sends customized text messages to each participant. We are testing two types of text messages: proactive texts that aim to promote initial program engagement, and reactive texts that aim to promote program re-engagement after lapses in visits. These text message campaigns will be rolled out in the campuses of three college partners with the Houston Food Bank: The University of Houston, The University of Houston Downtown, and Houston City College.

This project will be conducted as a randomized controlled trial. The study will be carried out in partnership with the Houston Food Bank (HFB), which will oversee participant enrollment, visit data collection, the sending of the text messages, and the receipt of data from the colleges. The LEO research team will serve as a third party evaluator, assisting with the initial build of the randomization procedure and conducting data analysis.

At the first step, each of the three schools will conduct their typical outreach and marketing for this program. The research team has no involvement in this, and neither does HFB aside from owning and managing the enrollment form and providing food to be distributed. As a part of enrollment, students will complete the enrollment form, which includes demographic information and a consent section. This consent section will not be changed by the research team; HFB will continue to collect participant consent as they have done in past semesters.

Consent:
Food Scholarship participants consent to (i) being contacted electronically and (ii) their data being used “to do research” upon program enrollment. Participant data encompasses two key aspects, (1) a participant’s market visit data (owned by HFB) and (2) a participant’s academic data (owned by each respective school). With these parameters, the research team was granted a waiver of the requirement for documentation of informed consent (45 CFR 46.117) for this text-message based RCT that presents no more than minimal risk. This is a benign behavioral intervention that each participant already agrees to upon enrolling in the program. They also consent to their data (visit and education) being used for research upon enrollment.

Randomization:
Upon successful submission of the enrollment form, the information will be sent via an automatic workflow from an internal HFB roster sheet where each student will be randomly assigned into one of 8 treatment arms, with a secondary randomization in place for students assigned to receive reactive texts. Participants will have no knowledge of their treatment status. Treatment status has no impact on a participant’s ability to receive free food at a market at the school or any other agency (food pantry) in the HFB network.

Texting Participants:
HFB is leveraging an automated texting software to facilitate the text message campaign. Each day, an automated workflow will run, linking a specific tab on the roster sheet to Catalyst. This tab will contain the necessary information for customized text messages to each participant, in addition to two columns that will tell Catalyst if a text message should be sent, and, if so, which version of the text. There will be 4 versions of the text messages, 1 version of the proactive message, and 3 of the reactive. The timing and content of the reactive texts will vary depending on how long it has been since the student’s last visit to any HFB agency, with each potential messaging sequence being randomly assigned to students. No texts will be sent after the last day of classes at each school. At each school this is defined as the day before finals start. We go into further detail regarding the text messages in the Pre-Analysis Plan.

Data Collection:
Data collection for baseline characteristics happens instantaneously upon completion of the enrollment form. Students will also be sent a welcome email from HFB containing the pre-program survey. This is HFB’s standard operating procedure and will not change for the purposes of the RCT.

Each time a participant visits a market in the Food Scholarship program or any other agency of HFB, that visit is recorded in HFB’s Link2Feed system. This system will integrate with the roster sheet, and via automation, the research team and HFB will be able to effectively track a participant’s visit history for the purposes of sending reactive texts. Additional details regarding a participant’s visit will be stored securely in HFB’s data warehouse and only shared with the research team in de-identified format for analysis purposes as determined necessary by HFB and LEO.

All participants will be sent the endline survey after completion of the semester. We will contact participants regarding the endline survey up to 5 times: two emails and three texts. Messaging will be identical for all participants, with the exception of the random subset offered payment. For this group, each message will contain an additional line about their compensatory offer.

HFB has existing MOUs with each school that outlines the transfer of student data. Students consent to the sharing of this data with HFB and third party researchers (the latter in de-identified form) when completing the FS enrollment form. Aggregate-level data transfers (e.g., average GPA among all FS enrollees) between the universities and HFB have occurred regularly for over a year.

We hope to receive the following information in de-identified format at the individual-level at semester’s conclusion: baseline cumulative GPA, baseline credit hours accumulated, baseline semester credit hours, end-of-semester GPA, end-of-semester credit hours, and student status for the following semester. In order to receive this, each University will need to agree to an updated MOU with HFB to cover the transfer of this data, as it encompasses more fields than previously agreed. Under this new agreement, the sharing of individual-level data with third party researchers would be permitted. If we are unable to receive individual-level data, we hope to receive aggregate statistics for the above variables at the treatment-arm-level. If Universities do not agree to either transfer method, we will rely on self-reported measures from the participants - these questions are included on the end-of-semester survey. The proposed updates to the MOUs have been drafted and sent to the Universities for initial review.

HFB and LEO have a data use agreement in place that covers program enrollment, visit data, baseline survey responses, endline survey responses. An addendum will be drafted once we have clarity on the possibility of receiving de-identified education data from HFB via the Universities.

2026-01-05

2026-06-15

Number of visits to Food Scholarship markets and other HFB agencies divided by the number of weeks of enrollment in the Food Scholarship program.

All students who enroll in the FS program for the first time between Jan 1, 2026 and May 4, 2026 will be randomized into one of our treatment arms. Because students enrolling earlier in the semester will have access to the food pantries for a longer period of time, our primary outcome is total visits / weeks enrolled. We will also analyze several other measures of visit frequency, including:
Total visits
Any visit
Any visit in the final month of the semester
Visits in the days immediately following the reactive text
Probability of visiting a market in the first, second, and third weeks following a text message

Academic outcomes such as GPA, enrollment status, and credit hours
Quality of life survey measures such a food security, financial security, mental health, and human flourishing

Note that any outcomes coming from the quality of life survey will only be observed for those completing the survey. In all such instances, we will use Heckman Selection models to account for potentially endogenous selection into survey completion. The random offering of a financial incentive to complete the survey will serve as an exclusion restriction.

Each of the three schools will conduct their typical outreach and marketing for this program. Students interested in the Food Scholarship program will complete an enrollment form, which includes reporting demographic information and completing a consent section. Upon successful submission of the enrollment form, the information will be sent instantaneously via an automatic workflow from an existing HFB spreadsheet into the roster spreadsheet for this study that will randomize each student into one of 8 treatment arms, with a secondary randomization in place for students assigned to receive reactive texts. We plan to compare outcomes between the treatment and control groups to assess the impact of the proactive and reactive text messages.

We will also send an endline survey to all enrolled students after completion of the semester. We will contact participants regarding the endline survey up to 5 times: two emails and three texts. Messaging will be identical for all participants, with the exception of the random subset offered payment. For this group, each message will contain an additional line about their compensatory offer. We are leveraging the random offering of compensation to avoid bias resulting from unequal survey response rates between treatment and control groups.

We go into further detail about the experimental design in the Pre-Analysis Plan.

Randomization will occur immediately upon program enrollment, when the information recorded in the intake form is recorded in the roster sheet. This sheet will be pre-populated with a list of integers from 1 through 8, each corresponding to one of the eight treatment arms. The primary randomization list is generated in blocks of 72 assignments with a fixed, pre-specified distribution: values 1 and 2 appear 12 times each, and values 3–8 appear 8 times each. Within every block, assignments are randomly ordered using a Fisher–Yates shuffle driven by a seed-based, deterministic pseudo-random number generator (Mulberry32), ensuring uniform randomization and full reproducibility. Independent seeds are generated and logged for the primary randomization and for message sequence randomization to ensure mathematical independence between procedures. This procedure is repeated independently for 160 blocks, producing 11,520 total assignment slots. For each slot, the assignment value, block ID, within-block order, and a unique run identifier are recorded. The resulting list is generated once at the start of the study and stored in the roster sheet. As students enroll, they are assigned to the treatment arm indicated by the integer in the row corresponding to their position in the roster. This design ensures each student has a 1 in 6 probability of being assigned to treatment arms 1 and 2, and a 1 in 9 probability of being assigned to treatment arms 3-8. We go into further detail regarding this design in the Pre-Analysis Plan.

In addition to randomizing text message and incentive receipt, we will also randomize the ordering of the 3 reactive text messages. There are 6 possible orderings of the three messages, so among the roughly 2044 participants set to be assigned to receive reactive messages, roughly 340 will be assigned to receive each ordering. The message sequence list is randomly generated separately in blocks of 30, each containing an equal number of values 1–6 (five of each). Values within each block are permuted using the same method described above, with a different independently-generated seed, ensuring statistical independence from the primary randomization. Message sequences are assigned conditionally: only participants randomized to receive reactive texts (random number ≥ 5) are assigned a message sequence from this pool. Participants not receiving reactive texts are assigned a value of 0 for message sequence. This conditional assignment ensures that the distribution of message sequences is balanced among those actually receiving a reactive text, rather than across the full sample. Students assigned to receive reactive texts will have an equal probability of being assigned to one of the six message sequences; we do not include weighting in this randomization procedure.

Individual

No

0

The sample will cover all students enrolled in The University of Houston, The University of Houston Downtown, and Houston City College, who enroll into the Food Scholarship program for the first time during our study period (1/1/2026 through 5/4/2026). Historic enrollment records provided by HFB show approximately 4,600 students enrolled into the program across the three locations during the Spring 2025 semester. Since we anticipate a similar number of students enrolling, our estimated sample size is 4,600.

There are 4 texting treatment arms: none, proactive, reactive, and both. All students enrolled in the program will be invited to complete an endline survey, and 1/2 will be randomly offered to receive incentives for survey completion and 1/2 will receive no such offer incentives. Thus, there are 8 treatment arms in total and the anticipated share of students in each treatment arm is as follows:

No proactive + no reactive + no incentive: 1/6
No proactive + no reactive + incentive: 1/6
Proactive + no reactive + no incentive: 1/9
Proactive + no reactive + incentive: 1/9
No proactive + reactive + no incentive: 1/9
No proactive + reactive + incentive: 1/9
Proactive + reactive + no incentive: 1/9
Proactive + reactive + incentive: 1/9

The total estimated sample size is 4,600 with 1533 receiving no proactive or reactive text, while 1022 will be in each of {P NR,NP R,P R}; thus, roughly 2044 will be assigned to receive a proactive text and 2044 assigned to receive a reactive text. Furthermore, roughly 2300 will be offered to receive an incentive to complete the endline survey, while the other 2300 will not.

Our power calculations use administrative visit data from Spring 2025 to construct control group means. Students made an average of 2.75 visits per semester (SD = 4.06). Taking into account the number of weeks enrolled, we see an average of .179 visits per weeks enrolled (SD = .245). The design assigns: Control (C): 1/3 of sample → 1,533 students Proactive only (T1): 2/9 of sample → 1,022 students Reactive only (T2): 2/9 of sample → 1,022 students Proactive + Reactive (T3): 2/9 of sample → 1,022 students Our initial model includes indicators for T1, T2, and T3, with 1,022 students contributing to each treatment–control comparison. If interaction effects are null, we drop T3 and pool T1 and T2, resulting in 2,044 students contributing to treatment–control comparisons. We therefore report power calculations for both scenarios. MDEs represent the minimum detectable difference in mean outcomes between each treatment arm and control. 1. Continuous Outcome: Number of Market Visits power twomeans 2.75, n1(1533) n2(1022) sd(4.06) alpha(0.05) power(0.8) MDE: 0.456 visits Interpretation: We can detect an increase from 2.75 visits (control mean) to 3.2 visits (treatment). This represents 16.6% of the control mean. power twomeans 2.75, n1(1533) n2(2044) sd(4.06) alpha(0.05) power(0.8) MDE: 0.384 visits Interpretation: We can detect an increase from 2.75 visits (control mean) to 3.1 visits (treatment). This represents 14% of the control mean. 2. Continuous Outcome: Number of Market Visits/Weeks Enrolled power twomeans .179, n1(1533) n2(1022) sd(.245) alpha(0.05) power(0.8) MDE: 0.028 visits/week enrolled Interpretation: We can detect an increase from .179 visits/weeks enrolled (control mean) to .207 visits/weeks enrolled (treatment). This represents 15.6% of the control mean. power twomeans .179, n1(1533) n2(2044) sd(.245) alpha(0.05) power(0.8) MDE: 0.023 visits/week enrolled Interpretation: We can detect an increase from .179 visits/weeks enrolled (control mean) to .202 visits/weeks enrolled (treatment). This represents 12.8% of the control mean.