x

NEW UPDATE: Completed trials may now upload and register supplementary documents (e.g. null results reports, populated pre-analysis plans, or post-trial results reports) in the Post Trial section under Reports, Papers, & Other Materials.
BIAS-NG: Monroe
Last registered on August 11, 2020

Pre-Trial

Trial Information
General Information
Title
BIAS-NG: Monroe
RCT ID
AEARCTR-0005451
Initial registration date
August 10, 2020
Last updated
August 11, 2020 9:27 AM EDT
Location(s)

This section is unavailable to the public. Use the button below to request access to this information.

Request Information
Primary Investigator
Affiliation
MDRC
Other Primary Investigator(s)
PI Affiliation
Harvard University
Additional Trial Information
Status
On going
Start date
2018-09-10
End date
2021-04-01
Secondary IDs
Abstract
The Behavioral Insights to Advance Self-Sufficiency-Next Generation (BIAS-NG) project, sponsored by the Office of Planning, Research, and Evaluation in the Administration for Children and Families (ACF), U.S. Department of Health and Human Services, uses principles of behavioral science in an effort to improve human services program design and outcomes. BIAS-NG partners with state and local agencies to diagnose behavioral barriers to program success, design interventions to address those barriers, and test the efficacy and cost efficiency of those behaviorally informed interventions relative to status quo service delivery. In Monroe County, NY, BIAS-NG worked with the Division of Financial Assistance (DFA), Monroe County Department of Social Services (MCDSS). DFA administers Monroe County’s Temporary Assistance for Needy Families (TANF) and Safety Net Assistance (SNA) programs.

Working closely with DFA, BIAS-NG identified behavioral barriers that program clients faced in successfully completing meetings and activities they were required to attend as a condition of public benefit receipt: the Employment Assessment (EA) meeting, the Work Experience Program (WEP) Enrollment meeting, and the WEP internship assignment. These meetings are designed to help clients fulfill their work requirements, avoid sanctions to their benefits, and, in the longer term, build work skills and increase their economic self-sufficiency. Increased attendance at these meetings and subsequent adherence to work activity requirements can also improve the county’s performance according to outcomes required for state and federal reporting.
As part of the “behavioral diagnosis and design process,” the research team worked to understand program challenges by: interviewing clients, staff, and administrators; observing orientation and individual meetings; and examining outreach materials. We conducted two tests to address the problems identified in the diagnosis of behavioral barriers to program success:
Test 1: A behavioral outreach intervention with behaviorally redesigned letters, a reminder magnet for clients to post in a visible place in their homes (for example, refrigerators), and text message reminders.
Test 2: A behaviorally informed WEP orientation to increase the rate of attendance at the WEP assignment.

For the impact evaluations for Test 1 and Test 2, clients were randomly assigned to the intervention or status quo services separately for each test. The behavioral outreach intervention (Test 1) was targeted to increase attendance at EA and WEP meetings. The behaviorally informed WEP orientation (Test 2) was targeted to increase attendance at WEP internship assignments. The impact evaluation will examine those outcomes on the full sample and on specified subgroups. In addition to the impact study, we are conducting accompanying implementation and cost analyses to document how the intervention was delivered and at what cost.
External Link(s)
Registration Citation
Citation
Katz, Larry and Clinton Key. 2020. "BIAS-NG: Monroe." AEA RCT Registry. August 11. https://doi.org/10.1257/rct.5451-1.0.
Sponsors & Partners

There are documents in this trial unavailable to the public. Use the button below to request access to this information.

Request Information
Experimental Details
Interventions
Intervention(s)
Working closely with DFA, BIAS-NG identified behavioral barriers that program clients faced in successfully completing meetings and activities they were required to attend as a condition of public benefits receipt: the EA meeting, the WEP Enrollment meeting, and the WEP internship assignment. These meetings are designed to help individuals fulfill their work requirements, avoid sanctions to their benefits, and, in the longer term, build work skills and increase their economic self-sufficiency. Increased attendance at these meetings and subsequent adherence to work activity requirements can also improve the county’s performance according to participation rate outcomes required for state and federal reporting. BIAS-NG implemented and tested two interventions to address the problems identified in behavioral diagnosis and design:
Test 1: A behavioral outreach intervention with behaviorally redesigned letters, a reminder magnet for clients to post in a visible place in their homes (for example, refrigerators), and text message reminders.
Test 2: A behaviorally informed WEP orientation to increase the rate of attendance at the WEP assignment.
Intervention Start Date
2018-09-10
Intervention End Date
2020-03-01
Primary Outcomes
Primary Outcomes (end points)
Test 1 will use the following primary outcomes:

a. Client attends EA meeting, measured at three time points: the initially scheduled meeting; within 30 days of random assignment at any EA meeting; and within 30 days of the initially scheduled EA meeting. This is a binary variable.
b. Client attends WEP meeting, measured at three time points: the initially scheduled meeting; within 30 days of random assignment at any WEP meeting; and within 30 days of the initially scheduled WEP meeting. This is a binary variable.

Test 2 will use the following primary outcomes:

a. Any attendance at the WEP assignment in the first week after the WEP meeting and within four weeks after the WEP meeting. This is a binary variable.
Primary Outcomes (explanation)
Secondary Outcomes
Secondary Outcomes (end points)
Sensitivity Analysis I for Outreach Test (Test 1)
Some portion of the sample, both treatment and control cases, did not have a cellular phone on record within the county or state data systems. As a result, this portion of the sample in the outreach test only received the behaviorally informed letter and magnet in the mail and did not receive the two text message reminders for the EA meeting or the WEP meeting. To address the fact that this subgroup did not receive the full intervention, we will conduct a sensitivity analysis of outcomes among only those individuals who have a phone number recorded in the county or state data systems. The rigor of this sensitivity analysis should be strong, as contact information for treatment and control group members should be of equal quality at the study baseline. This analysis will answer more specifically the question of the full behaviorally informed outreach intervention (two rounds of texts plus mailings) on attendance at EA and WEP meetings, and subsequently on attendance at the WEP assignment, in contrast to the main analysis, which will examine the impact of the intervention regardless of whether or not an individual could receive text messages.

Sensitivity Analysis II for Outreach Test (Test 1)
Some portion of the sample, both treatment and control individuals, were randomly assigned into the Test 2 sample prior to their random assignment into Test 1. We will conduct a sensitivity analysis of whether results differ for this sample compared to the full Test 1 sample.

Sensitivity Analysis III for Outreach Test (Test 1)
The behavioral outreach test originally targeted only TANF clients. However, declines in the County’s TANF caseload resulted in the County scheduling fewer TANF clients than anticipated for EA and WEP meetings during the early months of the test. The target group of the outreach test was subsequently expanded to include County Safety Net Assistance (SNA) clients. In Monroe County and throughout New York State, SNA provides aid to several categories of low-income individuals and families who are not eligible for TANF. This includes a combination of families, single adults, childless couples, and children. The income eligibility rules for SNA and TANF are similar, with both programs providing the same level of assistance based on household size. Some individuals on SNA receive cash assistance, while others receive non-cash assistance such as a payment made directly to the client’s landlord or vouchers provided to utility companies. To address the possibility that the SNA non-cash assistance clients may respond differently to the intervention, we will conduct a sensitivity analysis of whether results differ for the TANF and SNA cash assistance sample compared to the full Test 1 sample.

Sensitivity Analysis I for Orientation Test (Test 2)
Some portion of the sample, both treatment and control individuals, are not in the Test 1 sample. We will conduct a sensitivity analysis of outcomes among only those individuals who are in the Test 1 sample. This analysis will answer whether results differ for this sample compared to the full Test 2 sample.

Sensitivity Analysis II for Orientation Test (Test 2)
To address the possibility that the SNA non-cash assistance clients may respond differently to the intervention, we will conduct a sensitivity analysis of whether results differ for the TANF and SNA cash assistance sample compared to the full Test 2 sample.

Test 1 will use the following exploratory outcomes:

1. Attendance at an EA meeting within 60 days of random assignment, to answer longer-term questions about any EA attendance.
2. Attendance at a WEP meeting within 60 days of random assignment, to answer longer-term questions about any WEP meeting attendance.
3. Sanction status, measured based on the existence of any instance of a WEP-related sanction or being referred to a WEP-related sanction within 30 and 60 days of random assignment.

Test 2 will use the following exploratory outcomes:

1. Attendance at the WEP assignment within eight weeks after the WEP meeting, to answer longer-term questions about WEP assignment attendance.
2. Sanction status, measured based on the existence of any instance of a WEP-related sanction or being referred to a WEP-related sanction within 30 and 60 days of the WEP meeting.

Subgroup Analysis – Test 1 and Test 2

The evaluation will investigate whether the interventions worked especially well or less well for particular subgroups. Data from the Monroe County IT system will be used to explore differences in attendance at the EA and WEP meetings and the presence of any sanctions. This evaluation will use a “split-sample” subgroup analysis, in which the full sample is divided into two or more mutually exclusive and exhaustive groups. Impacts are estimated for each group separately. In addition to determining whether the interventions had statistically significant effects for each subgroup, we will calculate whether impacts differ significantly across subgroups.

We expect to see some variation in effects across certain subgroups. The subgroup analysis will be exploratory, will be measured for outcomes across both tests, and will utilize the same data sources we are using to answer our research questions. The following three subgroups will be analyzed:

a. TANF versus SNA cases: explore possible differences based on the nature of the population under each program, given that both populations were included in our tests.
For approximately 18% of the sample in Test 1 and 9% of the sample in Test 2, the current case type was not available in the data at baseline. For those missing current case type at baseline, case type is imputed using prior or later data. Overall, for the Test 1 sample, approximately 95% use case type at the time of random assignment or within 90 days. Overall, for the Test 2 sample, approximately 93% use case type at the time of random assignment or within 90 days. 4.5% of the Test 2 sample are missing case type.
b. Repeaters (clients who participated in WEP within the past year) versus those who are not repeating: repeaters have past experiences that may impact their decisions and behaviors compared to those who have less recent experience with the system.

c. Test 1 random assignment: Once they attended the EA meeting, those randomly assigned to either the Test 1 behavioral outreach or standard outreach conditions had an equal probability of assignment to either the Test 2 behaviorally informed orientation or the standard orientation. The groups should have equal probability of random assignment in Test 2. Test 1 randomization assignment and Test 2 randomization assignment are strictly independent. This subgroup analysis will determine if those who receive the behaviorally informed outreach are affected the same way by the behaviorally informed orientation as those who received the standard outreach, measured by attendance at the WEP assignment (within the first week and within the first four weeks).
Secondary Outcomes (explanation)
Experimental Design
Experimental Design
For the impact evaluations for Test 1 and Test 2, clients will be randomly assigned to the intervention or status quo services separately for each test. The behavioral outreach intervention (Test 1) is targeted to increase attendance at EA and WEP meetings. The behaviorally informed WEP orientation (Test 2) will be targeted to increase attendance at WEP assignments. The impact evaluation will examine those outcomes on the full sample and on specified subgroups. In addition to the impact study, we will conduct accompanying implementation and cost analyses to document how the intervention was delivered and at what cost.
Experimental Design Details
Not available
Randomization Method
For Test 1, computerized randomization was done by MDRC.
Randomization occurred at the level of the individual. If a second adult was randomly assigned, this adult received the same assignment as the first adult.

For Test 2, computerized randomization was done by MDRC. Randomization occurred at the level of the WEP Enrollment Meeting Date.
Randomization Unit
Individual-level randomization for the EA and WEP outreach tests. Randomization occurred by day of the orientation test, with analysis done at the individual level.
Was the treatment clustered?
Yes
Experiment Characteristics
Sample size: planned number of clusters
For Test 2, 239 days of orientation
Sample size: planned number of observations
For Test 1: 3,575 individuals For Test 2: 1,173 individuals
Sample size (or number of clusters) by treatment arms
For Test 1: 1,788 individuals treatment, 1,788 individuals control
For Test 2: 587 individuals treatment, 587 individuals control
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
The below estimates use a two-tailed test with a significance level of 10% and assume 80% power. Test 1: With an estimated 3,575 sample members invited to an EA meeting, the smallest true effect MDRC could detect of the behavioral outreach would be a change from the baseline EA attendance rate of 50.0% to 54.2% and a change in the WEP meeting attendance rate from the baseline WEP meeting attendance rate of 25.5% to 29.1%. Test 2: For the WEP orientation test, with an estimated 1,173 sample members attending, MDRC could detect a change in the WEP assignment attendance rate of 15.9 percentage point change, from 68% to 83.9%.
IRB
INSTITUTIONAL REVIEW BOARDS (IRBs)
IRB Name
MDRC IRB, New York, NY
IRB Approval Date
2019-07-19
IRB Approval Number
MDRC IRB #0003522
Analysis Plan

There are documents in this trial unavailable to the public. Use the button below to request access to this information.

Request Information