BIAS-NG: Monroe

Working closely with DFA, BIAS-NG identified behavioral barriers that program clients faced in successfully completing meetings and activities they were required to attend as a condition of public benefits receipt: the EA meeting, the WEP Enrollment meeting, and the WEP internship assignment. These meetings are designed to help individuals fulfill their work requirements, avoid sanctions to their benefits, and, in the longer term, build work skills and increase their economic self-sufficiency. Increased attendance at these meetings and subsequent adherence to work activity requirements can also improve the county’s performance according to participation rate outcomes required for state and federal reporting. BIAS-NG implemented and tested two interventions to address the problems identified in behavioral diagnosis and design:
Test 1: A behavioral outreach intervention with behaviorally redesigned letters, a reminder magnet for clients to post in a visible place in their homes (for example, refrigerators), and text message reminders.
Test 2: A behaviorally informed WEP orientation to increase the rate of attendance at the WEP assignment.

We conducted two tests to address the problems identified in the diagnosis of behavioral barriers to program success.
Test 1
1a. Behavioral outreach intervention for EA meeting attendance. The control group received a standard EA referral letter. This behavioral outreach intervention consisted of a redesigned EA letter plus the addition of a magnet and text message reminders.
i. Behaviorally redesigned EA referral letters: These letters were mailed to clients approximately two weeks before the scheduled EA meeting date. They addressed the behavioral bottlenecks of inattention, cognitive load, misinformation, negative affect, and planning failures. The letter focused on: simplifying information; making decision-relevant information (such as benefits and consequences) salient; highlighting social proof of attendance benefits; and leveraging the endowed progress effect and implementation prompts. The EA meeting duration was described as two hours rather than a “full-day assessment” as in the standard letter. This aspect of the letter addressed hassle factors and included planning prompts.
ii. Reminder magnet: A 3’’x 4’’ reminder magnet was mailed to clients along with the EA referral letter. The magnet included the MCDSS logo and the text “We are expecting you!” This magnet was intended to address inattention and prospective memory.
iii. Text message reminders: One text message was sent to clients informing them about the EA meeting one week before the scheduled date. The message included the date and time of the meeting, which clients could click to create an appointment to their phone or computer calendars. The message also included the meeting address, which clients could click to show on a mapping app, obtain directions, and determine the amount of time necessary for travel. This text may have served either as a reminder or as an initial notice, for those who did not receive or open the EA referral letter. This text was intended to address inattention and prospective memory. Two business days before the meeting date, a second text message was sent to clients as a reminder, to further address prospective memory.

1b. Behavioral outreach intervention for WEP meeting attendance. After completing the EA, some clients are deemed eligible for the WEP internship assignments and are required to attend a WEP Enrollment meeting, which consists of an orientation followed by a one-on-one placement meeting into an internship. At the end of the EA meeting, both the control and the intervention group clients assigned a WEP enrollment meeting, received a letter specifying the scheduled meeting in person. The intervention group clients received an additional, behaviorally informed letter and text message reminders.
i. Behaviorally informed WEP reminder letters: These letters were mailed to those intervention group 1a clients who attended an EA meeting and were assigned a WEP orientation. This means that the intervention group clients received two letters, but the mailed letter served as a behaviorally informed reminder to attend. These letters were mailed to clients seven days before their scheduled WEP meeting. The mailed letter addressed the behavioral bottlenecks of inattention, cognitive load, misinformation, negative affect, and planning failures. The letter focused on: simplifying information; making decision-relevant information (such as benefits and consequences) salient; and leveraging descriptive norms, the endowed progress effect, and implementation prompts.
ii. Text message reminders: One text message was sent to intervention group clients informing them about the WEP meeting one week before the scheduled date. The message contained the date and time of the meeting, which clients could click to create an appointment on their calendars. The message also contained the address for the meeting, which clients could click to show on a mapping app, obtain directions, and determine the amount of time necessary for travel. This text message served as a reminder for those who may have misplaced the letter and/or had not read the letter. This message was intended to address inattention and prospective memory. Two business days before the meeting date, a second text message was sent to clients as a reminder, to further address prospective memory.

Test 2:
Behaviorally informed WEP orientation intervention for WEP assignment attendance. The control group clients received the standard WEP orientation, designed primarily to be a high-level, one-way lecture. The intervention group clients received a behaviorally revised orientation, designed to address the behavioral bottlenecks uncovered in diagnosis of cognitive load, limited attention during the orientation, low motivation to engage in attending the WEP assignment, and implementation barriers (such as the intention-action gap) to go to the WEP assignment.
i. Behaviorally revised WEP orientation presentation: The revised presentation reduced and simplified the content and addressed certain challenges to attendance at the WEP assignment. The PowerPoint presentation was formatted using an aesthetically simple and pleasing design. The introduction to the WEP was brief, with added content detailing why clients often fail to comply with their WEP assignment. The revised presentation also used three specific proactive strategies: 1) addressing reasons individuals do not complete WEP internship assignments; 2) celebrating and identifying client successes using testimonials from past WEP; and 3) highlighting information on potential assistance (e.g., transportation, childcare, work attire) clients might want to plan for and access.
ii. Group discussion: The standard one-way presentation was changed to a group discussion format. The presenters encouraged client engagement and interaction, directing questions and strategies to the clients themselves. To achieve this, the presenters used discussion prompts and other techniques designed to increase engagement (e.g., approaching clients before responding to their questions, repeating or rephrasing questions to draw other clients’ attention to the questions and possible solutions, encouraging group responses to the questions). The revised presentation delivery method was intended to address the bottleneck of lack of motivation to engage in the orientation and, subsequently, the WEP assignment by increasing group interaction.

2018-09-10

2020-03-01

Test 1 will use the following primary outcomes:

a. Client attends EA meeting, measured at three time points: the initially scheduled meeting; within 30 days of random assignment at any EA meeting; and within 30 days of the initially scheduled EA meeting. This is a binary variable.
b. Client attends WEP meeting, measured at three time points: the initially scheduled meeting; within 30 days of random assignment at any WEP meeting; and within 30 days of the initially scheduled WEP meeting. This is a binary variable.

Test 2 will use the following primary outcomes:

a. Any attendance at the WEP assignment in the first week after the WEP meeting and within four weeks after the WEP meeting. This is a binary variable.

Sensitivity Analysis I for Outreach Test (Test 1)
Some portion of the sample, both treatment and control cases, did not have a cellular phone on record within the county or state data systems. As a result, this portion of the sample in the outreach test only received the behaviorally informed letter and magnet in the mail and did not receive the two text message reminders for the EA meeting or the WEP meeting. To address the fact that this subgroup did not receive the full intervention, we will conduct a sensitivity analysis of outcomes among only those individuals who have a phone number recorded in the county or state data systems. The rigor of this sensitivity analysis should be strong, as contact information for treatment and control group members should be of equal quality at the study baseline. This analysis will answer more specifically the question of the full behaviorally informed outreach intervention (two rounds of texts plus mailings) on attendance at EA and WEP meetings, and subsequently on attendance at the WEP assignment, in contrast to the main analysis, which will examine the impact of the intervention regardless of whether or not an individual could receive text messages.

Sensitivity Analysis II for Outreach Test (Test 1)
Some portion of the sample, both treatment and control individuals, were randomly assigned into the Test 2 sample prior to their random assignment into Test 1. We will conduct a sensitivity analysis of whether results differ for this sample compared to the full Test 1 sample.

Sensitivity Analysis III for Outreach Test (Test 1)
The behavioral outreach test originally targeted only TANF clients. However, declines in the County’s TANF caseload resulted in the County scheduling fewer TANF clients than anticipated for EA and WEP meetings during the early months of the test. The target group of the outreach test was subsequently expanded to include County Safety Net Assistance (SNA) clients. In Monroe County and throughout New York State, SNA provides aid to several categories of low-income individuals and families who are not eligible for TANF. This includes a combination of families, single adults, childless couples, and children. The income eligibility rules for SNA and TANF are similar, with both programs providing the same level of assistance based on household size. Some individuals on SNA receive cash assistance, while others receive non-cash assistance such as a payment made directly to the client’s landlord or vouchers provided to utility companies. To address the possibility that the SNA non-cash assistance clients may respond differently to the intervention, we will conduct a sensitivity analysis of whether results differ for the TANF and SNA cash assistance sample compared to the full Test 1 sample.

Sensitivity Analysis I for Orientation Test (Test 2)
Some portion of the sample, both treatment and control individuals, are not in the Test 1 sample. We will conduct a sensitivity analysis of outcomes among only those individuals who are in the Test 1 sample. This analysis will answer whether results differ for this sample compared to the full Test 2 sample.

Sensitivity Analysis II for Orientation Test (Test 2)
To address the possibility that the SNA non-cash assistance clients may respond differently to the intervention, we will conduct a sensitivity analysis of whether results differ for the TANF and SNA cash assistance sample compared to the full Test 2 sample.

Test 1 will use the following exploratory outcomes:

1. Attendance at an EA meeting within 60 days of random assignment, to answer longer-term questions about any EA attendance.
2. Attendance at a WEP meeting within 60 days of random assignment, to answer longer-term questions about any WEP meeting attendance.
3. Sanction status, measured based on the existence of any instance of a WEP-related sanction or being referred to a WEP-related sanction within 30 and 60 days of random assignment.

Test 2 will use the following exploratory outcomes:

1. Attendance at the WEP assignment within eight weeks after the WEP meeting, to answer longer-term questions about WEP assignment attendance.
2. Sanction status, measured based on the existence of any instance of a WEP-related sanction or being referred to a WEP-related sanction within 30 and 60 days of the WEP meeting.

Subgroup Analysis – Test 1 and Test 2

The evaluation will investigate whether the interventions worked especially well or less well for particular subgroups. Data from the Monroe County IT system will be used to explore differences in attendance at the EA and WEP meetings and the presence of any sanctions. This evaluation will use a “split-sample” subgroup analysis, in which the full sample is divided into two or more mutually exclusive and exhaustive groups. Impacts are estimated for each group separately. In addition to determining whether the interventions had statistically significant effects for each subgroup, we will calculate whether impacts differ significantly across subgroups.

We expect to see some variation in effects across certain subgroups. The subgroup analysis will be exploratory, will be measured for outcomes across both tests, and will utilize the same data sources we are using to answer our research questions. The following three subgroups will be analyzed:

a. TANF versus SNA cases: explore possible differences based on the nature of the population under each program, given that both populations were included in our tests.
For approximately 18% of the sample in Test 1 and 9% of the sample in Test 2, the current case type was not available in the data at baseline. For those missing current case type at baseline, case type is imputed using prior or later data. Overall, for the Test 1 sample, approximately 95% use case type at the time of random assignment or within 90 days. Overall, for the Test 2 sample, approximately 93% use case type at the time of random assignment or within 90 days. 4.5% of the Test 2 sample are missing case type.
b. Repeaters (clients who participated in WEP within the past year) versus those who are not repeating: repeaters have past experiences that may impact their decisions and behaviors compared to those who have less recent experience with the system.

c. Test 1 random assignment: Once they attended the EA meeting, those randomly assigned to either the Test 1 behavioral outreach or standard outreach conditions had an equal probability of assignment to either the Test 2 behaviorally informed orientation or the standard orientation. The groups should have equal probability of random assignment in Test 2. Test 1 randomization assignment and Test 2 randomization assignment are strictly independent. This subgroup analysis will determine if those who receive the behaviorally informed outreach are affected the same way by the behaviorally informed orientation as those who received the standard outreach, measured by attendance at the WEP assignment (within the first week and within the first four weeks).

For the impact evaluations for Test 1 and Test 2, clients will be randomly assigned to the intervention or status quo services separately for each test. The behavioral outreach intervention (Test 1) is targeted to increase attendance at EA and WEP meetings. The behaviorally informed WEP orientation (Test 2) will be targeted to increase attendance at WEP assignments. The impact evaluation will examine those outcomes on the full sample and on specified subgroups. In addition to the impact study, we will conduct accompanying implementation and cost analyses to document how the intervention was delivered and at what cost.

Interventions and Random Assignment Strategy

1) Test 1a: Behavioral outreach intervention test for EA meeting attendance
MDRC team generated a random assignment sequence file and shared it with the MCDSS information technology team. This sequence was merged with the current MCCSS EA meeting booking database. As clients became eligible for an EA meeting date, client records appeared on an MCDSS database. Using the random assignment sequence, clients were flagged as 0 (control; standard outreach) or 1 (treatment; behavioral outreach). MCDSS staff received a pop-up if a client was flagged as Treatment, which guided staff to print and mail the behaviorally informed EA referral letter plus the magnet. The list of intervention group clients was then sent to the text messaging system on a daily basis. If multiple individuals from the same household were assigned EA meeting dates, that entire household was assigned to the same group (control or treatment). Clients who rescheduled or repeated a meeting stayed in their originally randomized group.

Throughout the random assignment process, MDRC monitored the process to ensure staff correctly adhered to the protocol.

2) Test 1b: Behavioral outreach intervention test for WEP meeting attendance
The random assignment status generated for the EA meeting outreach intervention was also used for this stage of Test 1. In other words, MDRC did not re-randomize clients. Caseworkers handed the standard WEP letters to any client they met during an EA meeting. For clients in the treatment group only, one MCDSS staff member mailed a behavioral WEP reminder letters seven days prior to their scheduled WEP meeting. Once client EA attendance was recorded in the database, the list of intervention group clients was then automatically sent to the text messaging system, to be manually pushed out by one MCDSS staff on a daily basis.

Throughout the random assignment process, MDRC monitored the process to ensure staff correctly adhered to the protocol.

3) Test 2: Behaviorally informed WEP orientation intervention test for WEP assignment attendance
MDRC randomly assigned WEP meeting days to either the standard or the behaviorally informed orientation. The MDRC team generated a random assignment sequence file and sent it to staff at RochesterWorks!, an MCDSS subcontractor. Using this sequence file, WEP orientation days for the test duration were specified as treatment or control days. Clients who attended WEP meetings on control days attended the standard orientation. Clients who attended WEP meetings on treatment days attended the behaviorally informed orientation.

MDRC closely monitored this system to ensure that random assignment worked as intended. RochesterWorks! staff provided the WEP orientation for both treatment and control groups.

Given concerns of spillover effects, RochesterWorks! designated distinct staff provide the behaviorally informed orientation and the standard control group orientation. However, due to a limited number of staff who lead the orientations, occasional illness or emergencies resulted in staff trained in one orientation method needing to substitute for staff trained in another. MDRC documented the extent to which this occurred.

For Test 1, computerized randomization was done by MDRC.
Randomization occurred at the level of the individual. If a second adult was randomly assigned, this adult received the same assignment as the first adult.

For Test 2, computerized randomization was done by MDRC. Randomization occurred at the level of the WEP Enrollment Meeting Date.

Individual-level randomization for the EA and WEP outreach tests. Randomization occurred by day of the orientation test, with analysis done at the individual level.

Yes

For Test 2, 239 days of orientation

For Test 1: 3,575 individuals For Test 2: 1,173 individuals

For Test 1: 1,788 individuals treatment, 1,788 individuals control
For Test 2: 587 individuals treatment, 587 individuals control

The below estimates use a two-tailed test with a significance level of 10% and assume 80% power. Test 1: With an estimated 3,575 sample members invited to an EA meeting, the smallest true effect MDRC could detect of the behavioral outreach would be a change from the baseline EA attendance rate of 50.0% to 54.2% and a change in the WEP meeting attendance rate from the baseline WEP meeting attendance rate of 25.5% to 29.1%. Test 2: For the WEP orientation test, with an estimated 1,173 sample members attending, MDRC could detect a change in the WEP assignment attendance rate of 15.9 percentage point change, from 68% to 83.9%.