Understanding the effects of text reminders on reducing churn in supplemental assistance programs

“Churning” is when eligible transitional benefit program participants fail to renew, temporarily dropping out of their program before reentering within four months. Churn is a widespread issue across transitional benefit programs, negatively impacting participants and administering agencies. This proposed intervention will assess whether a low-cost scalable texting program can reduce churn for the Supplemental Nutrition Assistance Program. Participants in one treatment group will receive text messages that simplify the interim report and recertification process with reminders, information, and direct links. A second treatment arm will receive reminders only, while a control group will experience the current procedures of the Massachusetts Department of Transitional Assistance. The text messages aim to reduce informational and behavioral barriers program participants face and thus lessen churn. We estimate an overall sample size around 60,000 over the course of three months. We will iterate three times during the project, testing timing of the texts and identifying heterogeneous effects. The primary outcome of interest is churn probability. We will also assess completing process steps on time rates, submitting complete forms or verification rates, interview completion rates (including as-scheduled and post-missed interview), approval and closure rates, and the average number of days that forms are submitted before deadlines.

For this intervention, we will randomly assign SNAP benefit participants who are due for their interim report or re-certification and are thus at risk of churning into groups. The first treatment group will receive text message reminders about key steps, a phone number that they can call to receive help, and hyperlinks to the website and mobile application where clients can submit their forms. This treatment addresses informational constraints such as lack of information on how and where to file the paperwork. It also addresses the behavioral barriers created by cognitive load by including easy links and holds client attention through reminders. A second treatment arm will receive reminders only. All text messages will include a note that recipients can “Text STOP to end messages.” The control group consists of participants receiving the standard letters and automated phone calls currently in effect.

This research design answers these key questions:
(1) can a text-messaging program reduce churn; and
(2) do reminders drive the effects or is a reduction in cognitive load through easy to implement steps necessary to reduce churn.

The comparison of the two treatments provides evidence on whether reminders alone are beneficial or whether easing the processes and reducing the cognitive load are necessary. We will run the intervention for three months and then iterate on the texts, aiming to identify particularly promising practices. The large sample and the ability to iterate will allow us to explore whether the time of day or day of the week for texts effect their impact, as was found for parent texting; what the optimal number of text messages is; and whether certain groups respond differentially including groups by age, location, children, work history, and race/ethnicity. Texting programs are unusually easy to implement with fidelity and we are using available data.

2022-02-17

2023-02-01

The primary outcome of interest is churn probability. We will also assess completing process steps on time rates, submitting complete forms or verification rates, interview completion rates (including as-scheduled and post-missed interview), approval and closure rates, and the average number of days that forms are submitted before deadlines.

For this intervention, we will randomly assign SNAP benefit participants who are due for their interim report or re-certification and are thus at risk of churning into groups. The first treatment group will receive text message reminders about key steps, a phone number that they can call to receive help, and hyperlinks to the website and mobile application where clients can submit their forms. This treatment addresses informational constraints such as lack of information on how and where to file the paperwork. It also addresses the behavioral barriers created by cognitive load by including easy links and holds client attention through reminders. A second treatment arm will receive reminders only. All text messages will include a note that recipients can “Text STOP to end messages.” The control group consists of participants receiving the standard letters and automated phone calls currently in effect.

This research design answers these key questions:
(1) can a text-messaging program reduce churn; and
(2) do reminders drive the effects or is a reduction in cognitive load through easy to implement steps necessary to reduce churn.

The comparison of the two treatments provides evidence on whether reminders alone are beneficial or whether easing the processes and reducing the cognitive load are necessary. We will run the intervention for three months and then iterate on the texts, aiming to identify particularly promising practices. The large sample and the ability to iterate will allow us to explore whether the time of day or day of the week for texts effect their impact, as was found for parent texting; what the optimal number of text messages is; and whether certain groups respond differentially including groups by age, location, children, work history, and race/ethnicity. Texting programs are unusually easy to implement with fidelity and we are using available data.

All SNAP recipients who are due for their interim report or re-certification and who have provided their phone number to DTA will be considered for this intervention. We have only limited information on the opt-out rates, but believe that 20 percent is a conservative estimate, given opt-out from other studies. While DTA’s caseload fluctuates, a conservative estimate of approximately 10,000 re-certification forms and 15,000 Interim Reports (IRs) will be due every month for SIMP-12 SNAP clients. With an opt out rate of 20 percent, we will have a monthly sample size of around 8,000 household heads (10,000 re-certification due x 80%=8,000) who will need to complete re-certification forms, and of 12,000 household heads (15,000 IRs due x 80%=12,000) who will need to complete an IR.

Given the nature of this project, we intend to treat the maximum number of eligible participants. We intend to first field this intervention for three months (February 2020 through May 2020) and estimate an overall sample size around 51,000 clients who DTA has permission to communicate with via text message. Using Optimal Design based on a single level person randomized trial for the calculations, we expect to be able to find acceptable minimal detectable effects sizes (MDE) with this sample. The MDE of a binary treatment comparison (C vs T1, C vs T2, or T1 vs T2) for a power of 80 percent, a significance level of 5 percent, and sample sizes of between 60 percent and a 20 percent opt out rate. We anticipate these opt-out rates are much higher than we will encounter with the intervention, since clients have already had the opportunity to opt-out of text message communication. In the worst expected case, with an opt-out rate of 60 percent (51,000*0.4=20,400), comparing two treatment groups (20,400*2/3=13,600) and an R2 of three percent, we will be able to detect effects of a size of 0.047 standard deviations (SD) or larger for any binary treatment comparison. In more likely scenarios, we will be able to detect effect sizes as low as 0.028 SDs. Given a historic average churn of 22 percent (SD=0.414), these effect sizes translate to detecting MDEs between 1.2-1.9 percentage points. Previous related studies find larger or similar effect sizes.

We will estimate the average treatment effect in a straightforward linear model:
Y_i=β_0+β_1 T_i^1+β_2 T_i^2+X_i β_3+ε_i where Y_i is the outcome of interest of household head i (submission of re-certification form or interim report, SNAP benefit renewal), and T_i^1 and T_i^2 are binary indicators for treatment groups. We will include participant background characteristics, X_i, to improve precision but they should not affect point estimates.

We will randomly assign SNAP benefit participants who are due for their interim report or re-certification and are thus at risk of churning into three groups using statistical software in the Annenberg Institute at Brown University office using secure computers:

(T1) The first treatment group will receive text message reminders about key steps, a phone number that they can call to receive help, and hyperlinks to the website and mobile application where clients can submit their forms.

(T2) A second treatment arm will receive text reminders only.

(C) A control group consisting of participants receiving the standard letters and automated phone calls currently in effect.

All SNAP recipients who are due for their interim report or re-certification and who have provided their phone number to DTA will be considered for this intervention.

We will randomly assign SNAP benefit participants who are due for their interim report or re-certification, and are thus at risk of churning, at the individual level.

No

Given the nature of this project, we intend to treat the maximum number of eligible participants. We intend to first field this intervention for three months (February 2020 through May 2020) and estimate an overall sample size around 51,000 clients who DTA has permission to communicate with via text message.

Our planned number of observations is the same as our planned number of clusters due to the fact that the designed is not clustered.Given this, we again intend to treat the maximum number of eligible participants. We intend to first field this intervention for three months (February 2020 through May 2020) and estimate an overall sample size around 51,000 clients who DTA has permission to communicate with via text message.

We aim to evenly split our randomized sample across Treatment arm 1/Treatment arm 2/Control arm, and estimate approximately 17,000 participants each in the control (C), reminder (T2), and reminder+action conditions (T1).

Using Optimal Design based on a single level person randomized trial for the calculations, we expect to be able to find acceptable minimal detectable effects sizes (MDE) with this sample. Our power calculation shows the MDE of a binary treatment comparison (C vs T1, C vs T2, or T1 vs T2) for a power of 80 percent, a significance level of five percent, and sample sizes of between a 60 percent and a 20 percent opt-out rate. We anticipate these opt-out rates are much higher than we will encounter with the intervention, since clients have already had the opportunity to opt-out from text message communication. The two lines on the graph, attached in the pre-analysis plan, represent R2s explained by covariates of three percent (as found in previous analyses using a limited set of covariates) and 15 percent. We present a range of R2s because we will have a full set of covariates available in our estimation, which will likely result in an R2 higher than three percent. In the worst expected case, with an opt-out rate of 60 percent ( 51,000 *0.4=20,400), comparing two treatment groups ( 20,400 *2/3= 13,600) and an R2 of three percent, we will be able to detect effects of a size of 0.047 standard deviations (SD) or larger for any binary treatment comparison. In more likely scenarios, we will be able to detect effect sizes as low as 0.028 SDs. Given a historic average churn of 22 percent (SD=0.414), these effect sizes translate to detecting MDEs between 1.2-1.9 percentage points.