Evaluating form modifications to make it easier to report income on benefits applications – Evidence from single income households

Last registered on March 19, 2024
View Trial History

Pre-Trial

Trial Information

General Information

Title
Evaluating form modifications to make it easier to report income on benefits applications – Evidence from single income households
RCT ID
AEARCTR-0013183
Initial registration date
March 12, 2024

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
March 19, 2024, 5:03 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Primary Investigator

Name
Blair Read
Affiliation
Office of Evaluation Sciences
Contact Primary Investigator

Other Primary Investigator(s)

PI Name
Mary Clair Turner
PI Affiliation
Office of Evaluation Sciences
Contact Investigator
PI Name
Stephanie Tepper
PI Affiliation
Office of Evaluation Sciences
Contact Investigator
PI Name
Jae Yeon Kim
PI Affiliation
Code for America
Contact Investigator

Additional Trial Information

Status
In development
Start date
2023-05-01
End date
2024-10-01
Keywords
Welfare
Additional Keywords
Social policy; Safety Net; Social Welfare Programs; Poverty alleviation
JEL code(s)
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
This project aims to improve applicants’ experiences with reporting their income on the Supplemental Nutrition Assistance Program (SNAP) application. The SNAP application requires applicants to provide information about their income in the past 30 days to assess eligibility and calculate benefits amounts. Additionally, applicants have to upload verification documents that confirm their responses on the application. This project uses an application digital assister, designed by Code For America (CfA), to help individuals in a large U.S. state complete the SNAP application. This evaluation examines which of three different sets of options for reporting income are more effective at encouraging applicants to complete the SNAP application. In the federal context, multiple benefits applications ask applicants to report their income, and administrative burdens in form completion can impede access to federal programs among eligible beneficiaries. This project also will build generalizable evidence regarding how to ask people about income that can be applied across forms, benefits applications, and surveys administered by the federal government that include modules on income reporting.
External Link(s)

Registration Citation

Citation
Kim, Jae Yeon et al. 2024. "Evaluating form modifications to make it easier to report income on benefits applications – Evidence from single income households." AEA RCT Registry. March 19. https://doi.org/10.1257/rct.13183-1.0
Sponsors & Partners
Experimental Details

Interventions

Intervention(s)
In this evaluation, SNAP applicants were randomized to see one of three sets of options for reporting their income. Each of the reporting choice sets displays a different set of income reporting options. Applicants in the control condition were asked to select between using their hourly wage to enter in earned income or entering the exact amount from their pay stubs. Applicants in Intervention Arm A were asked to select between using their hourly wage or using their yearly salary. Applicants in Intervention Arm B were asked to select between using their hourly wage, using their yearly salary, or entering the exact amount from their pay stubs. Applicants in all conditions also were given the option to “give an estimate,” where they were provided with a text box to enter in their earned income for the last 30 days.

The income reporting module occurs after applicants have answered basic questions about themselves, including the county they live in, their disability status, their student status, their citizenship status, and if they receive income support in the form of SSI or disability payments. Applicants are then asked if they are self-employed, if there are multiple jobs in their household, and if they have stable or variable income. Applicants who are self-employed, have a single job in their household, and have stable income are included in the evaluation.
Intervention Start Date
2023-05-08
Intervention End Date
2023-06-23

Primary Outcomes

Primary Outcomes (end points)
Application completion
Primary Outcomes (explanation)
The primary outcome of interest is application completion, a binary measure equal to 1 if the applicant entered in a signature on the last page of the application and hit “sign and submit application” and 0 otherwise.

Secondary Outcomes

Secondary Outcomes (end points)
Income reporting option selected (descriptive analysis)
Income module completion
Uploading of any SNAP verification documents
Predicted SNAP eligibility
Total time spent on the application
Total time spent to submit the application
Total amount of income reported
Screen of application drop-off
Secondary Outcomes (explanation)
Secondary outcomes are listed above.

Experimental Design

Experimental Design
In this evaluation, SNAP applicants were randomized to see one of three sets of options for reporting their income. Each of the reporting choice sets displays a different set of income reporting options. Applicants in the control condition were asked to select between using their hourly wage to enter in earned income or entering the exact amount from their pay stubs. Applicants in Intervention Arm A were asked to select between using their hourly wage or using their yearly salary. Applicants in Intervention Arm B were asked to select between using their hourly wage, using their yearly salary, or entering the exact amount from their pay stubs. Applicants in all conditions also were given the option to “give an estimate,” where they were provided with a text box to enter in their earned income for the last 30 days.

The income reporting module occurs after applicants have answered basic questions about themselves, including the county they live in, their disability status, their student status, their citizenship status, and if they receive income support in the form of SSI or disability payments. Applicants are then asked if they are self-employed, if there are multiple jobs in their household, and if they have stable or variable income. Applicants who are self-employed, have a single job in their household, and have stable income are included in the evaluation.
Experimental Design Details
Not available
Randomization Method
Applicants were randomized using Vanity, an Experiment Driven Development platform. Each eligible applicant had an equal chance of being assigned to each of the intervention arms. See more about Vanity randomization at https://vanity.labnotes.org/ab_testing.html.
Randomization Unit
Units are randomized at the level of individual application session.
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
N/A
Sample size: planned number of observations
~ 33,000
Sample size (or number of clusters) by treatment arms
We anticipate a sample size of ~33,000 where 11,000 applicants are in the control, 11,000 are in Intervention Arm A, and 11,000 are in intervention Arm B.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
The minimum detectable effect size for our primary outcome is approximately 1.75 percentage points with a sample of approximately 33,000. This power analysis assumes 80% power and a baseline submission rate of 43%, which is based on historical data from the digital assister. It uses a Bonferroni correction to adjust for multiple comparisons to account for three pairwise comparisons (comparing the control arm and two intervention arms to each other) for one outcome variable.
Supporting Documents and Materials
IRB

Institutional Review Boards (IRBs)

IRB Name
University of Maryland College Park (UMCP), College Park, MD
IRB Approval Date
2023-10-17
IRB Approval Number
2116542-1
Analysis Plan

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information