For this intervention, we will randomly assign SNAP benefit participants who are due for their interim report or re-certification and are thus at risk of churning into groups. The first treatment group will receive text message reminders about key steps, a phone number that they can call to receive help, and hyperlinks to the website and mobile application where clients can submit their forms. This treatment addresses informational constraints such as lack of information on how and where to file the paperwork. It also addresses the behavioral barriers created by cognitive load by including easy links and holds client attention through reminders. A second treatment arm will receive reminders only. All text messages will include a note that recipients can “Text STOP to end messages.” The control group consists of participants receiving the standard letters and automated phone calls currently in effect.
This research design answers these key questions:
(1) can a text-messaging program reduce churn; and
(2) do reminders drive the effects or is a reduction in cognitive load through easy to implement steps necessary to reduce churn.
The comparison of the two treatments provides evidence on whether reminders alone are beneficial or whether easing the processes and reducing the cognitive load are necessary. We will run the intervention for three months and then iterate on the texts, aiming to identify particularly promising practices. The large sample and the ability to iterate will allow us to explore whether the time of day or day of the week for texts effect their impact, as was found for parent texting; what the optimal number of text messages is; and whether certain groups respond differentially including groups by age, location, children, work history, and race/ethnicity. Texting programs are unusually easy to implement with fidelity and we are using available data.
All SNAP recipients who are due for their interim report or re-certification and who have provided their phone number to DTA will be considered for this intervention. We have only limited information on the opt-out rates, but believe that 20 percent is a conservative estimate, given opt-out from other studies. While DTA’s caseload fluctuates, a conservative estimate of approximately 10,000 re-certification forms and 15,000 Interim Reports (IRs) will be due every month for SIMP-12 SNAP clients. With an opt out rate of 20 percent, we will have a monthly sample size of around 8,000 household heads (10,000 re-certification due x 80%=8,000) who will need to complete re-certification forms, and of 12,000 household heads (15,000 IRs due x 80%=12,000) who will need to complete an IR.
Given the nature of this project, we intend to treat the maximum number of eligible participants. We intend to first field this intervention for three months (February 2020 through May 2020) and estimate an overall sample size around 51,000 clients who DTA has permission to communicate with via text message. Using Optimal Design based on a single level person randomized trial for the calculations, we expect to be able to find acceptable minimal detectable effects sizes (MDE) with this sample. The MDE of a binary treatment comparison (C vs T1, C vs T2, or T1 vs T2) for a power of 80 percent, a significance level of 5 percent, and sample sizes of between 60 percent and a 20 percent opt out rate. We anticipate these opt-out rates are much higher than we will encounter with the intervention, since clients have already had the opportunity to opt-out of text message communication. In the worst expected case, with an opt-out rate of 60 percent (51,000*0.4=20,400), comparing two treatment groups (20,400*2/3=13,600) and an R2 of three percent, we will be able to detect effects of a size of 0.047 standard deviations (SD) or larger for any binary treatment comparison. In more likely scenarios, we will be able to detect effect sizes as low as 0.028 SDs. Given a historic average churn of 22 percent (SD=0.414), these effect sizes translate to detecting MDEs between 1.2-1.9 percentage points. Previous related studies find larger or similar effect sizes.
We will estimate the average treatment effect in a straightforward linear model:
Y_i=β_0+β_1 T_i^1+β_2 T_i^2+X_i β_3+ε_i where Y_i is the outcome of interest of household head i (submission of re-certification form or interim report, SNAP benefit renewal), and T_i^1 and T_i^2 are binary indicators for treatment groups. We will include participant background characteristics, X_i, to improve precision but they should not affect point estimates.