Behavioral Interventions to Advance Self Sufficiency – Next Generation (BIAS-NG): Los Angeles County, Temporary Assistance for Needy Families

Last registered on November 30, 2022

Pre-Trial

General Information

Title

RCT ID

AEARCTR-0010539

Initial registration date

November 29, 2022

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published

November 30, 2022, 4:59 PM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Country

United States of America

Region

California

Primary Investigator

Name

Clinton Key

Affiliation

MDRC

Contact Primary Investigator

Other Primary Investigator(s)

PI Name

Lawrence Katz

PI Affiliation

Harvard University

Contact Investigator

Additional Trial Information

Status

On going

Start date

2019-02-18

End date

2023-10-31

Keywords

Behavior, Welfare

Additional Keywords

JEL code(s)

H53 (Government Expenditures and Welfare Programs) D91 (Role and Effects of Psychological, Emotional, Social, and Cognitive Factors on Decision Making) I38 (Government Policy • Provision and Effects of Welfare Programs) H75 (State and Local Government • Health • Education • Welfare • Public Pensions • Intergovernmental Relations)

Secondary IDs

Prior work

This trial does not extend or rely on any prior RCTs.

Abstract

The Behavioral Interventions to Advance Self-Sufficiency-Next Generation (BIAS-NG) project, sponsored by the Office of Planning, Research, and Evaluation in the Administration for Children and Families (ACF), U.S. Department of Health and Human Services, uses principles of behavioral science to try to improve human services program design and outcomes. Behavioral science is an interdisciplinary field that incorporates psychology, economics, and other social sciences to provide insight into how people process information, make decisions, and take actions. BIAS-NG partners with state and local human services agencies to diagnose behaviorally-based barriers to program success, design interventions to address those barriers, and test the efficacy and cost efficiency of those behaviorally informed interventions relative to status quo service delivery.

For this study, BIAS-NG worked with the Los Angeles County Department of Public Social Service (DPSS). DPSS manages the Los Angeles County Temporary Assistance to Needy Families (TANF) program called California Work Opportunities and Responsibilities to Kids (CalWORKs). California’s TANF policies mandate that adult CalWORKs recipients participate in 20-35 hours per week of approved work activities, or welfare-to-work activities, as a condition of receiving benefits. If CalWORKs recipients are required to participate in welfare-to-work activities, they must also attend required meetings and report the hours they spent on these activities.

In Los Angeles County, the welfare-to-work activities are administered through the Greater Avenues for Independence (GAIN) program which is also overseen by DPSS. All adults in Los Angeles County who receive public assistance benefits through CalWORKs are required to participate in GAIN unless they meet criteria for an exemption, although those who are “exempt” may participate voluntarily. The GAIN program offers a variety of welfare-to-work activities that aim to support participants in finding sustainable work. One of these activities is Job Club, which is the focus of this BIAS-NG study. Job Club is a four-week program designed to support participants with building skills they need to find and obtain a job, such as developing a resume and interviewing techniques. Job Club participants then apply these skills to searching for a job during the program period.

When BIAS-NG and DPSS began working together, DPSS staff identified the challenge of low participant engagement in GAIN activities and set a goal of increasing engagement. Low engagement in GAIN activities is a challenge for both staff and participants. If a GAIN participant does not comply with the program requirements, GAIN program staff place the participant in a non-compliance status, initiating a process that may lead to the participant being sanctioned and having their benefits reduced. Having a participant enter non-compliance status is a very time-consuming process for both staff and participants. When a participant is placed in non-compliance status, the GAIN Service Worker (GSW) who manages the participant’s case must attempt to contact the participant multiple times, and if the participant does not respond to attempts to schedule a meeting, the GSW will make a visit to the participant's home. The GSW and participant must have a meeting to determine whether a participant had “good cause” for not complying with requirements before sanctions are issued. This whole process often takes two months to complete.

Additionally, participants can miss out on substantial benefits if sanctioned. In 2018, for one-person households, sanctioning reduced monthly benefits by as much as $355. For two- or three-person households, which often include children whose benefits are not reduced by sanctions, benefits were reduced by as much as $222 or $137, respectively. Of GAIN participants not exempted from the work requirement, 36 percent were in sanction status as of August 2018.

Working closely with DPSS, the research team identified key points in GAIN’s program processes (such as meetings, transitions, or activities) where participants did not meet GAIN program requirements or stopped participating in GAIN en masse. We refer to these points as “drop-off points.” The analysis of these drop-off points led the team to focus on attendance at Job Club. Of participants who were referred to Job Club in June 2017, only 38 percent started Job Club within 30 days of referral. Activity attendance is a problem across other GAIN activities as well.

The research team conducted interviews with GAIN staff and participants and used the information they shared to hypothesize several behavioral barriers that may be impacting Job Club attendance. These included participants not understanding GAIN program requirements, sanctions not being salient enough to achieve their purpose of motivating participants to comply with GAIN requirements, hassles associated with arranging child care and transportation, negative associations with GAIN programs, and having other life stressors that make attending Job Club a less pressing concern.

In response to these behavioral challenges, DPSS and the research team developed a text messaging intervention, further described in the Intervention section below. The goal of the texting intervention is to increase participation in Job Club, thereby reducing the incidence of sanctions due to failure to comply with GAIN requirements. In addition to initial attendance, the intervention was designed to increase the percentage of participants who complete Job Club successfully, meaning they find employment within the four weeks of Job Club and stop attending, or they complete all four weeks of Job Club. To capture these two goals, the primary outcomes will be attendance at Job Club in the first 30 days after referral, attendance at Job Club within the first 60 days after referral, and entrance into non-compliance status within 60 days of referral. The first two measures correspond with key drop-off points: 1) between the point someone is assigned to Job Club and the scheduled Job Club start date and 2) between the scheduled Job Club start date and completion.

For the impact evaluation, clients were randomly assigned to receive the behaviorally informed text messages (program group) or not (control group). The point of random assignment was when clients were referred, or assigned, to Job Club. The evaluation will compare outcomes of the program group to the control group for the full sample and several sub-groups. In addition to the impact study, BIAS-NG is conducting accompanying implementation and cost analyses to document how the intervention was delivered and at what cost.

External Link(s)

Registration Citation

Citation

Katz, Lawrence and Clinton Key. 2022. "Behavioral Interventions to Advance Self Sufficiency – Next Generation (BIAS-NG): Los Angeles County, Temporary Assistance for Needy Families ." AEA RCT Registry. November 30. https://doi.org/10.1257/rct.10539-1.0

Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Experimental Details

Interventions

Intervention(s)

The BIAS-NG intervention aims to increase participation in Job Club, thereby reducing non-compliance with welfare-to-work requirements and reducing sanctioning, by using text message reminders with embedded behavioral messages. Such messages have been shown to improve attendance to healthcare appointments and appearances to court summonses. A study among TANF recipients in one Colorado county found that similar messages delivered by phone had significant positive effects on attendance to initial welfare-to-work meetings. Together, these findings suggest that behaviorally informed text messages are a promising tool for encouraging TANF recipients to attend welfare-to-work activities.

Intervention (Hidden)

For those program participants who agreed to participate in the study and signed an informed consent document to receive text messages, the GSWs entered the participants’ information such as first name, ID number, phone number, and some details about their Job Club assignment into the random assignment (RA) tool. The RA tool then randomly assigned participants into either the program or control group. The text messaging intervention included twelve sets of text messages. Each message set was comprised of two to five individual texts intended to be read together to form a single message for the participant. The first six message sets were sent before participants started Job Club. These “pre-Job Club” messages were sent to all participants assigned to the program group. They were sent to participants beginning on the day of random assignment and ending on the scheduled Job Club start date. The messages included personalized reminders and behaviorally informed language encouraging attendance at Job Club. The other six message sets were sent to program group participants after their scheduled Job Club start date, and the message sets that were sent to each participant depended on whether the person attended Job Club on their scheduled start date. Participants received up to four sets of weekly messages if they were attending and up to two sets of weekly messages if they were not attending.

GAIN participants referred to Job Club and randomly assigned to the control group received the business-as-usual condition: a paper form with the date, time, and location of Job Club during their meeting with the GSW. Control group members did not receive any intervention text messages. GSWs completed follow-up with participants in both the program and control groups as they normally would.

Text messages sent to program group members were generated by the research team using templates and information that GSWs entered into the RA tool. Once the text messages were generated, they were then sent via secure file transfer protocol to LA County DPSS and sent to participants using an automated texting platform already in use by DPSS for another program. Messages were one-way because the platform did not have the capacity to receive any responses from recipients. Rather than solicit text responses, the messages included the GSW’s phone number and prompted the participant to call the GSW with any questions.

Using Job Club attendance or non-attendance to tailor the text message sets that were sent after Job Club started required GAIN staff to complete several steps. Once per week, one GSW per Job Club location checked the Job Club attendance of participants who were assigned to Job Club at that location and who had been randomized to receive text messages. The GSW then entered the attendance information into the RA tool. That GSW was expected to enter the attendance information into the RA tool on or before the day texts were sent each week.

Participants who had attended that week received texts with encouragement to persist through Job Club. Participants who had not attended that week received texts containing reminders to contact their GSW as soon as possible to help them avoid being placed in a non-compliance status. The GSW who updated the RA tool also indicated if a participant left Job Club because they got a job, which stopped future messages about Job Club from being sent to that participant.

The intervention was implemented in several phases. First, the intervention was piloted with people assigned to Job Club in one of the seven GAIN regions in Los Angeles County. The pilot period was February 18 to September 20, 2019, and it included a sample of 192 participants across the control and program groups. The pilot began with sending only the six pre-Job Club message sets that were sent between the date of random assignment and the scheduled Job Club start date. During the time when the program group was only sent pre-Job Club messages, the pilot enrolled 144 of the initial 192 in the sample across the control and program groups. Near the end of the pilot period, the complete intervention with all twelve message sets was implemented for participants with a scheduled Job Club start date on or after August 12, 2019. During that time, the other 48 pilot participants were enrolled into the study across the control and program groups. From that point onwards, participants assigned to the program group received the intervention message sets leading up to Job Club as well as message sets sent after a participant’s scheduled Job Club start date.

After the pilot was complete, the full-scale study began on September 23, 2019. DPSS expanded the intervention into the remaining six GAIN regions in Los Angeles County. Additionally, while the pilot study only included messages in English, the full-scale intervention also included messages translated to Spanish for those who preferred receiving messages in Spanish. Enrollment into the study ended on March 16, 2020.

The 48 participants in the pilot sample who were enrolled once the complete intervention was implemented were included in the full sample for most of the study’s outcomes and all pilot participants were included in the sample for one outcome related to attendance on the first day of Job Club.

Intervention Start Date

2019-02-18

Intervention End Date

2020-03-16

Primary Outcomes

Primary Outcomes (end points)

i. % of individuals who participated in Job Club within 30 days of Random Assignment (RA).
ii. % of individuals who participated in Job Club within 60 days of RA.
iii. % of individuals who entered non-compliance within 60 days of RA.

Primary Outcomes (explanation)

The primary outcomes will be created using administrative data provided by DPSS. For Job Club participation measures, the data provide indications for Job Club activity, whether that activity was started and completed, the initial scheduled date, and the dates of starting and completing the activity. For the non-compliance measure, the data provide the status of participation in the welfare-to-work program including an indication for status category of "Non-compliance" and the start date of that status category.

Secondary Outcomes

Secondary Outcomes (end points)

i. % of individuals who started Job Club on the first scheduled date.
ii. % of individuals who completed Job Club within 90 days of RA.
iii. % of individuals who entered non-compliance within 30 days of RA.
iv. % of individuals who were sanctioned within 90 days of RA.
v. % of individuals who were approved for Child Care Assistance within 30 days of RA.
vi. % of individuals who were approved for Child Care Assistance within 60 days of RA.

Secondary Outcomes (explanation)

The secondary outcomes will be created using administrative data provided by DPSS. For Job Club participation measures, the data provide indications for Job Club activity, whether that activity was started and completed, the initial scheduled date, and the dates of starting and completing the activity. For non-compliance and sanction measures, the data provide the status of participation in the welfare-to-work program including indications for status categories of "Sanction" and "Non-compliance" and the start date of that status category. Child Care Assistance measures are included as a secondary outcome because one set of text messages provided participants with contact information for their child care provider to support them with planning for child care while attending Job Club. For these Child Care Assistance measures, the data provide indications for receiving Child Care Assistance and the start date of the assistance.

Subgroup Analysis
The evaluation will investigate whether the intervention worked especially well for particular subgroups of families. In this evaluation a “split-sample” subgroup analyses will be used in which the full sample is divided into two or more mutually exclusive and exhaustive groups. Impacts are estimated for each group separately. In addition to determining whether the intervention had statistically significant effects for each subgroup, analyses will determine whether impacts differ significantly across subgroups.

We expect to see some variation in effects across certain subgroups. The subgroup analysis will be exploratory, and will utilize the same data sources we are using to answer our research questions. Impacts on the following three subgroups will be analyzed:
• Age of participant (e.g., under age 25/age 25 and older): Younger people may have stronger digital literacy and may be more comfortable communicating via text message. Younger and older individuals may also see different value in welfare-to-work activities based on different life stages. For example, older individuals may have had more experiences building the job search skills, like resume building and interviewing techniques, but they may be newer skills for younger participants. Therefore, we may expect to see higher impacts for younger participants.
• Age of youngest child (e.g., under age 5/age 5 and older): As child care may be a barrier to attending Job Club, participants with children under age five may be more likely to experience the structural barrier to attendance of not having child care relative to those with older children who do not need child care. This could make participants with young children less likely to respond to the behavioral intervention addressing those barriers than those with school-age children.
• Months on TANF (e.g., fewer than six months/greater than six months): Participants who have been on TANF for longer than six months have likely already attended some GAIN activities. With that prior experience, their behavior may be less responsive to this type of behavioral intervention than participants who are new to GAIN.

Experimental Design

The impact evaluation uses a randomized controlled trial design. Study participants have an equal (50/50) chance of being assigned to the program or control groups.

The study sample includes Los Angeles County CalWORKs participants who are age 18 or older, speak English or Spanish, do not have a confidential case file or current domestic violence issues, are required to participate in welfare-to-work activities, and attend an in-person meeting with a GSW during which they are referred to Job Club. The sample includes participants who are both new and not new to GAIN. Study enrollment could occur at different points after enrolling in GAIN:
• During the initial appraisal meeting, after referral from CalWORKs.
• After completing an assigned activity and coming in to sign a new welfare-to-work plan.
• At a “cause determination” meeting, in which participants attend to correct an instance of non-compliance and sign a compliance plan.

The steps of the random assignment process were as follows:
• Step 1: During the meeting between the GSW and participant in which participants were referred to Job Club, the GSW explained to the participant that the county was interested in sending text messages to participants but was trying it out to determine whether it was helpful to participants. The GSW briefly explained the study, and if the participant was interested in receiving text messages, he or she signed an informed consent form agreeing to receive text messages if assigned to the program group. The GSW confirmed that GAIN had the right phone number to send the participant text messages. GSWs also emphasized that regardless of whether a participant received text messages, they were required to attend Job Club on their assigned date. Both control and program group participants received paper forms that included this information.
• Step 2: After the participant left the meeting, the GSW logged into the RA tool and entered the participant’s GAIN region, first name, and case ID.
• Step 3: The GSW was then prompted to enter in a few additional pieces of information including the participant’s phone number, confirmation that the assigned activity was Job Club, activity start date and time, activity end date and time, and activity location. The GSW then submitted the information to the RA tool, which randomly assigned the participant to the program or control group.
• Step 4: If the participant was assigned to the program group, the RA tool used the submitted information to populate text message templates and determine which messages to send each day. This report was automatically uploaded to LA County’s secure file transfer protocol each day. The IT Division’s system automatically downloaded the report and sent out the messages as texts.
• Step 5: A GSW co-located at each Job Club logged into the RA tool once a week to indicate if all the participants in the program group assigned to attend Job Club at the location where they worked were: (1) still attending Job Club, (2) had dropped out of Job Club or were no shows, or (3) had left Job Club because they found a job. This information was used to populate the appropriate text message templates sent after the participant’s scheduled Job Club start date. These messages were sent in the same way as the pre-Job Club messages. Note: this step was not needed at the start of the pilot, and was added beginning the week of August 12, 2019, when the complete intervention was first implemented.
• Ongoing: If the participant’s information changed (e.g., the start date of Job Club was delayed), the GSW logged into the RA tool and updated the information.

For two-parent cases, the second parent to be randomized was automatically assigned to the same research group as their partner to reduce potential contamination of information in the text messages sent to program group members being shared with control group members. For cases that were subject to randomization multiple times (e.g., a participant was assigned to a new GSW, who randomized the participant again during a cause determination meeting), the case retained its original research group assignment.

Experimental Design Details

Randomization Method

MDRC random assignment web tool

Randomization Unit

Randomization occurred at the level of the case. Each individual was randomly assigned, and if another adult in the same case was later also randomly assigned, this adult received the same assignment as the first adult.

Was the treatment clustered?

Experiment Characteristics

Sample size: planned number of clusters

n/a

Sample size: planned number of observations

Intended target sample size: 3,300 individuals Realized sample size: 1,205 individuals (1,157 individuals from the full scale and 48 individuals from the pilot) are in the sample from the period of time that program group participants received the complete intervention. There are an additional 144 individuals from the pilot sample, who were enrolled when program group participants received only the six message sets leading up to Job Club. Most outcomes will be measured using the 1,205 individuals who were randomly assigned during the period when the program group received the complete intervention. The secondary outcome, % of individuals who started Job Club on the first scheduled date, will be measured using the full realized sample size of 1,349 individuals. As a sensitivity analysis, this outcome will also be measured using the 1,205 individuals who were randomly assigned during the period when the program group received the complete intervention.

Sample size (or number of clusters) by treatment arms

Intended: 1,650 individuals treatment, 1,650 individuals control

Realized: Depending on the outcome being measured, 1,349 or 1,205 individuals, half in treatment and half in control.

Minimum detectable effect size for main outcomes (accounting for sample design and clustering)

The below estimates use a two-tailed test with a significance level of 10% and assume 80% power with a sample size of 1,205 individuals. The smallest true effect we could detect from the intervention is: An effect of 7.0 percentage points on the rate of individuals who participated in Job Club within 30 days of RA. An effect of 7.1 percentage points on the rate of individuals who participated in Job Club within 60 days of RA. An effect of 7.0 percentage points on the rate of individuals who entered non-compliance within 60 days of RA.

Supporting Documents and Materials

IRB