BIAS-NG: Monroe

Last registered on August 11, 2020

Pre-Trial

Trial Information

General Information

Title
BIAS-NG: Monroe
RCT ID
AEARCTR-0005451
Initial registration date
August 10, 2020

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
August 11, 2020, 9:27 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Primary Investigator

Affiliation
MDRC

Other Primary Investigator(s)

PI Affiliation
Harvard University

Additional Trial Information

Status
On going
Start date
2018-09-10
End date
2021-04-01
Secondary IDs
Abstract
The Behavioral Insights to Advance Self-Sufficiency-Next Generation (BIAS-NG) project, sponsored by the Office of Planning, Research, and Evaluation in the Administration for Children and Families (ACF), U.S. Department of Health and Human Services, uses principles of behavioral science in an effort to improve human services program design and outcomes. BIAS-NG partners with state and local agencies to diagnose behavioral barriers to program success, design interventions to address those barriers, and test the efficacy and cost efficiency of those behaviorally informed interventions relative to status quo service delivery. In Monroe County, NY, BIAS-NG worked with the Division of Financial Assistance (DFA), Monroe County Department of Social Services (MCDSS). DFA administers Monroe County’s Temporary Assistance for Needy Families (TANF) and Safety Net Assistance (SNA) programs.

Working closely with DFA, BIAS-NG identified behavioral barriers that program clients faced in successfully completing meetings and activities they were required to attend as a condition of public benefit receipt: the Employment Assessment (EA) meeting, the Work Experience Program (WEP) Enrollment meeting, and the WEP internship assignment. These meetings are designed to help clients fulfill their work requirements, avoid sanctions to their benefits, and, in the longer term, build work skills and increase their economic self-sufficiency. Increased attendance at these meetings and subsequent adherence to work activity requirements can also improve the county’s performance according to outcomes required for state and federal reporting.
As part of the “behavioral diagnosis and design process,” the research team worked to understand program challenges by: interviewing clients, staff, and administrators; observing orientation and individual meetings; and examining outreach materials. We conducted two tests to address the problems identified in the diagnosis of behavioral barriers to program success:
Test 1: A behavioral outreach intervention with behaviorally redesigned letters, a reminder magnet for clients to post in a visible place in their homes (for example, refrigerators), and text message reminders.
Test 2: A behaviorally informed WEP orientation to increase the rate of attendance at the WEP assignment.

For the impact evaluations for Test 1 and Test 2, clients were randomly assigned to the intervention or status quo services separately for each test. The behavioral outreach intervention (Test 1) was targeted to increase attendance at EA and WEP meetings. The behaviorally informed WEP orientation (Test 2) was targeted to increase attendance at WEP internship assignments. The impact evaluation will examine those outcomes on the full sample and on specified subgroups. In addition to the impact study, we are conducting accompanying implementation and cost analyses to document how the intervention was delivered and at what cost.
External Link(s)

Registration Citation

Citation
Katz, Larry and Clinton Key. 2020. "BIAS-NG: Monroe." AEA RCT Registry. August 11. https://doi.org/10.1257/rct.5451-1.0
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
Working closely with DFA, BIAS-NG identified behavioral barriers that program clients faced in successfully completing meetings and activities they were required to attend as a condition of public benefits receipt: the EA meeting, the WEP Enrollment meeting, and the WEP internship assignment. These meetings are designed to help individuals fulfill their work requirements, avoid sanctions to their benefits, and, in the longer term, build work skills and increase their economic self-sufficiency. Increased attendance at these meetings and subsequent adherence to work activity requirements can also improve the county’s performance according to participation rate outcomes required for state and federal reporting. BIAS-NG implemented and tested two interventions to address the problems identified in behavioral diagnosis and design:
Test 1: A behavioral outreach intervention with behaviorally redesigned letters, a reminder magnet for clients to post in a visible place in their homes (for example, refrigerators), and text message reminders.
Test 2: A behaviorally informed WEP orientation to increase the rate of attendance at the WEP assignment.
Intervention Start Date
2018-09-10
Intervention End Date
2020-03-01

Primary Outcomes

Primary Outcomes (end points)
Test 1 will use the following primary outcomes:

a. Client attends EA meeting, measured at three time points: the initially scheduled meeting; within 30 days of random assignment at any EA meeting; and within 30 days of the initially scheduled EA meeting. This is a binary variable.
b. Client attends WEP meeting, measured at three time points: the initially scheduled meeting; within 30 days of random assignment at any WEP meeting; and within 30 days of the initially scheduled WEP meeting. This is a binary variable.

Test 2 will use the following primary outcomes:

a. Any attendance at the WEP assignment in the first week after the WEP meeting and within four weeks after the WEP meeting. This is a binary variable.
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Sensitivity Analysis I for Outreach Test (Test 1)
Some portion of the sample, both treatment and control cases, did not have a cellular phone on record within the county or state data systems. As a result, this portion of the sample in the outreach test only received the behaviorally informed letter and magnet in the mail and did not receive the two text message reminders for the EA meeting or the WEP meeting. To address the fact that this subgroup did not receive the full intervention, we will conduct a sensitivity analysis of outcomes among only those individuals who have a phone number recorded in the county or state data systems. The rigor of this sensitivity analysis should be strong, as contact information for treatment and control group members should be of equal quality at the study baseline. This analysis will answer more specifically the question of the full behaviorally informed outreach intervention (two rounds of texts plus mailings) on attendance at EA and WEP meetings, and subsequently on attendance at the WEP assignment, in contrast to the main analysis, which will examine the impact of the intervention regardless of whether or not an individual could receive text messages.

Sensitivity Analysis II for Outreach Test (Test 1)
Some portion of the sample, both treatment and control individuals, were randomly assigned into the Test 2 sample prior to their random assignment into Test 1. We will conduct a sensitivity analysis of whether results differ for this sample compared to the full Test 1 sample.

Sensitivity Analysis III for Outreach Test (Test 1)
The behavioral outreach test originally targeted only TANF clients. However, declines in the County’s TANF caseload resulted in the County scheduling fewer TANF clients than anticipated for EA and WEP meetings during the early months of the test. The target group of the outreach test was subsequently expanded to include County Safety Net Assistance (SNA) clients. In Monroe County and throughout New York State, SNA provides aid to several categories of low-income individuals and families who are not eligible for TANF. This includes a combination of families, single adults, childless couples, and children. The income eligibility rules for SNA and TANF are similar, with both programs providing the same level of assistance based on household size. Some individuals on SNA receive cash assistance, while others receive non-cash assistance such as a payment made directly to the client’s landlord or vouchers provided to utility companies. To address the possibility that the SNA non-cash assistance clients may respond differently to the intervention, we will conduct a sensitivity analysis of whether results differ for the TANF and SNA cash assistance sample compared to the full Test 1 sample.

Sensitivity Analysis I for Orientation Test (Test 2)
Some portion of the sample, both treatment and control individuals, are not in the Test 1 sample. We will conduct a sensitivity analysis of outcomes among only those individuals who are in the Test 1 sample. This analysis will answer whether results differ for this sample compared to the full Test 2 sample.

Sensitivity Analysis II for Orientation Test (Test 2)
To address the possibility that the SNA non-cash assistance clients may respond differently to the intervention, we will conduct a sensitivity analysis of whether results differ for the TANF and SNA cash assistance sample compared to the full Test 2 sample.

Test 1 will use the following exploratory outcomes:

1. Attendance at an EA meeting within 60 days of random assignment, to answer longer-term questions about any EA attendance.
2. Attendance at a WEP meeting within 60 days of random assignment, to answer longer-term questions about any WEP meeting attendance.
3. Sanction status, measured based on the existence of any instance of a WEP-related sanction or being referred to a WEP-related sanction within 30 and 60 days of random assignment.

Test 2 will use the following exploratory outcomes:

1. Attendance at the WEP assignment within eight weeks after the WEP meeting, to answer longer-term questions about WEP assignment attendance.
2. Sanction status, measured based on the existence of any instance of a WEP-related sanction or being referred to a WEP-related sanction within 30 and 60 days of the WEP meeting.

Subgroup Analysis – Test 1 and Test 2

The evaluation will investigate whether the interventions worked especially well or less well for particular subgroups. Data from the Monroe County IT system will be used to explore differences in attendance at the EA and WEP meetings and the presence of any sanctions. This evaluation will use a “split-sample” subgroup analysis, in which the full sample is divided into two or more mutually exclusive and exhaustive groups. Impacts are estimated for each group separately. In addition to determining whether the interventions had statistically significant effects for each subgroup, we will calculate whether impacts differ significantly across subgroups.

We expect to see some variation in effects across certain subgroups. The subgroup analysis will be exploratory, will be measured for outcomes across both tests, and will utilize the same data sources we are using to answer our research questions. The following three subgroups will be analyzed:

a. TANF versus SNA cases: explore possible differences based on the nature of the population under each program, given that both populations were included in our tests.
For approximately 18% of the sample in Test 1 and 9% of the sample in Test 2, the current case type was not available in the data at baseline. For those missing current case type at baseline, case type is imputed using prior or later data. Overall, for the Test 1 sample, approximately 95% use case type at the time of random assignment or within 90 days. Overall, for the Test 2 sample, approximately 93% use case type at the time of random assignment or within 90 days. 4.5% of the Test 2 sample are missing case type.
b. Repeaters (clients who participated in WEP within the past year) versus those who are not repeating: repeaters have past experiences that may impact their decisions and behaviors compared to those who have less recent experience with the system.

c. Test 1 random assignment: Once they attended the EA meeting, those randomly assigned to either the Test 1 behavioral outreach or standard outreach conditions had an equal probability of assignment to either the Test 2 behaviorally informed orientation or the standard orientation. The groups should have equal probability of random assignment in Test 2. Test 1 randomization assignment and Test 2 randomization assignment are strictly independent. This subgroup analysis will determine if those who receive the behaviorally informed outreach are affected the same way by the behaviorally informed orientation as those who received the standard outreach, measured by attendance at the WEP assignment (within the first week and within the first four weeks).
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
For the impact evaluations for Test 1 and Test 2, clients will be randomly assigned to the intervention or status quo services separately for each test. The behavioral outreach intervention (Test 1) is targeted to increase attendance at EA and WEP meetings. The behaviorally informed WEP orientation (Test 2) will be targeted to increase attendance at WEP assignments. The impact evaluation will examine those outcomes on the full sample and on specified subgroups. In addition to the impact study, we will conduct accompanying implementation and cost analyses to document how the intervention was delivered and at what cost.
Experimental Design Details
Interventions and Random Assignment Strategy

1) Test 1a: Behavioral outreach intervention test for EA meeting attendance
MDRC team generated a random assignment sequence file and shared it with the MCDSS information technology team. This sequence was merged with the current MCCSS EA meeting booking database. As clients became eligible for an EA meeting date, client records appeared on an MCDSS database. Using the random assignment sequence, clients were flagged as 0 (control; standard outreach) or 1 (treatment; behavioral outreach). MCDSS staff received a pop-up if a client was flagged as Treatment, which guided staff to print and mail the behaviorally informed EA referral letter plus the magnet. The list of intervention group clients was then sent to the text messaging system on a daily basis. If multiple individuals from the same household were assigned EA meeting dates, that entire household was assigned to the same group (control or treatment). Clients who rescheduled or repeated a meeting stayed in their originally randomized group.

Throughout the random assignment process, MDRC monitored the process to ensure staff correctly adhered to the protocol.

2) Test 1b: Behavioral outreach intervention test for WEP meeting attendance
The random assignment status generated for the EA meeting outreach intervention was also used for this stage of Test 1. In other words, MDRC did not re-randomize clients. Caseworkers handed the standard WEP letters to any client they met during an EA meeting. For clients in the treatment group only, one MCDSS staff member mailed a behavioral WEP reminder letters seven days prior to their scheduled WEP meeting. Once client EA attendance was recorded in the database, the list of intervention group clients was then automatically sent to the text messaging system, to be manually pushed out by one MCDSS staff on a daily basis.

Throughout the random assignment process, MDRC monitored the process to ensure staff correctly adhered to the protocol.

3) Test 2: Behaviorally informed WEP orientation intervention test for WEP assignment attendance
MDRC randomly assigned WEP meeting days to either the standard or the behaviorally informed orientation. The MDRC team generated a random assignment sequence file and sent it to staff at RochesterWorks!, an MCDSS subcontractor. Using this sequence file, WEP orientation days for the test duration were specified as treatment or control days. Clients who attended WEP meetings on control days attended the standard orientation. Clients who attended WEP meetings on treatment days attended the behaviorally informed orientation.

MDRC closely monitored this system to ensure that random assignment worked as intended. RochesterWorks! staff provided the WEP orientation for both treatment and control groups.

Given concerns of spillover effects, RochesterWorks! designated distinct staff provide the behaviorally informed orientation and the standard control group orientation. However, due to a limited number of staff who lead the orientations, occasional illness or emergencies resulted in staff trained in one orientation method needing to substitute for staff trained in another. MDRC documented the extent to which this occurred.

Randomization Method
For Test 1, computerized randomization was done by MDRC.
Randomization occurred at the level of the individual. If a second adult was randomly assigned, this adult received the same assignment as the first adult.

For Test 2, computerized randomization was done by MDRC. Randomization occurred at the level of the WEP Enrollment Meeting Date.
Randomization Unit
Individual-level randomization for the EA and WEP outreach tests. Randomization occurred by day of the orientation test, with analysis done at the individual level.
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
For Test 2, 239 days of orientation
Sample size: planned number of observations
For Test 1: 3,575 individuals For Test 2: 1,173 individuals
Sample size (or number of clusters) by treatment arms
For Test 1: 1,788 individuals treatment, 1,788 individuals control
For Test 2: 587 individuals treatment, 587 individuals control
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
The below estimates use a two-tailed test with a significance level of 10% and assume 80% power. Test 1: With an estimated 3,575 sample members invited to an EA meeting, the smallest true effect MDRC could detect of the behavioral outreach would be a change from the baseline EA attendance rate of 50.0% to 54.2% and a change in the WEP meeting attendance rate from the baseline WEP meeting attendance rate of 25.5% to 29.1%. Test 2: For the WEP orientation test, with an estimated 1,173 sample members attending, MDRC could detect a change in the WEP assignment attendance rate of 15.9 percentage point change, from 68% to 83.9%.
IRB

Institutional Review Boards (IRBs)

IRB Name
MDRC IRB, New York, NY
IRB Approval Date
2019-07-19
IRB Approval Number
MDRC IRB #0003522
Analysis Plan

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials