Behavioral Interventions to Advance Self Sufficiency – Next Generation (BIAS-NG): Los Angeles County, Child Welfare

Last registered on January 09, 2024

View Trial History

Pre-Trial

Trial Information

General Information

Title

Behavioral Interventions to Advance Self Sufficiency – Next Generation (BIAS-NG): Los Angeles County, Child Welfare

RCT ID

AEARCTR-0012781

Initial registration date

January 05, 2024

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published

January 09, 2024, 11:49 AM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Country

United States of America

Region

California

Primary Investigator

Name

Caitlin Anzelone

Affiliation

MDRC

Contact Primary Investigator

Other Primary Investigator(s)

PI Name

Lawrence Katz

PI Affiliation

Harvard University

Contact Investigator

Additional Trial Information

Status

On going

Start date

2022-04-27

End date

2025-03-31

Keywords

Behavior, Welfare

Additional Keywords

JEL code(s)

H53 (Government Expenditures and Welfare Programs) D91 (Role and Effects of Psychological, Emotional, Social, and Cognitive Factors on Decision Making) I38 (Government Policy • Provision and Effects of Welfare Programs) H75 (State and Local Government • Health • Education • Welfare • Public Pensions • Intergovernmental Relations)

Secondary IDs

Prior work

This trial does not extend or rely on any prior RCTs.

Abstract

The Behavioral Interventions to Advance Self-Sufficiency-Next Generation (BIAS-NG) project is supported by the Office of Planning, Research, and Evaluation in the Administration for Children and Families, U.S. Department of Health and Human Services. BIAS-NG aims to make human services programs work better for the people receiving services by reshaping program processes using lessons from behavioral science, an interdisciplinary field that incorporates psychology, economics, and other social sciences to provide insight into how people process information, make decisions, and take action. BIAS-NG partners with state and local agencies to identify a challenge to address, investigate its possible causes, design a behaviorally-informed intervention to address the causes, and test the efficacy and cost efficiency of the intervention relative to status quo service delivery. For this study, BIAS-NG worked with the Los Angeles County Department of Children and Family Services (DCFS) to use insights from behavioral science to design and test two interventions intended to improve the online application process for applicants to become foster or adoptive parents, who DCFS refers to as “resource families.”

Based on input from DCFS, BIAS-NG focused efforts on better understanding challenges in the resource family application process for applicants recruited from the community to foster or adopt children in the child welfare system to whom they are not related. Their applications are considered “community resource family applications.” Because the caseworkers from DCFS who approve these applicants tend to prioritize emergency placement cases, the approval process tends to be longer for community families.

The administration of the community resource family application process includes two primary phases for DCFS. First, the Outreach and Recruitment Unit (ORU), comprised of about ten caseworkers, support community resource family applicants through the initial steps of the application process. In this stage, most potential applicants create an account in county’s online application portal, operated through Binti, a software provider for child welfare agencies, and complete an online orientation. They can then sign and submit a Resource Family Approval (RFA) initial application form, which officially begins the application process. After completing the initial application form, families must take a series of actions to trigger next steps in the full application process, such as “LiveScan” fingerprinting, submitting additional forms, and enrolling in pre-approval training (collectively “Phase I” application steps). Once these steps are complete, the application moves to a second phase, in which a RFA approval caseworker is assigned who takes over the case to complete home visits and a psycho-social assessment.

While investigating DCFS’s processes, the research team identified a problem of community resource family applicants dropping out of the full application process after signing and submitting their RFA initial application form, contributing to a low approval rate. DCFS leadership expressed concern about applicant attrition for two reasons: (1) some qualified applicants who could be approved do not make it to the end of the process; and (2) staff spend a lot of time working with applicants who are not ready or qualified to be community resource parents, which is not an efficient use of staff resources.

The research team used administrative data about community resource family applicants to further understand the scope of this problem and quantified drop-off points, or key moments when applicants dropped out of the application process in large numbers. Data showed applicants left the application process at each stage, but the biggest drop-off point was shortly after applicants “sign” (submit) the initial application form. More than 59% of community resource family applicants were withdrawn from the application process by DCFS staff after they signed their initial application form. Staff withdraw applicants either at an applicant’s request or because applicants are non-responsive to follow up. Of the applicants that were withdrawn after signing an initial application form, 32% of them were withdrawn within one month of signing. This analysis of the quantitative data combined with the qualitative impressions from DCFS staff support a contention that some unqualified and unprepared families initially apply to be community resource parents and then quickly withdraw, and some qualified families find the application process challenging to navigate.

The research team conducted interviews with DCFS staff and community resource family applicants to inform hypotheses on the behavioral barriers that may hinder applicants in completing the application steps. These included that applicants do not know how to meet requirements, or have difficulty knowing what the most appropriate decision to make is regarding those requirements; applicants miss or do not fully process information about application steps or eligibility; applicants take more time to complete the application process than the county would expect; and applicants face multiple process hassles.

In response to these behavioral challenges, DCFS and the research team developed two interventions to be included in the Binti online application portal: a readiness assessment and a collection of planning tools. These are described further in the Intervention section. The goal of the readiness assessment is to help potential community resource family applicants make a fully informed decision about whether they are ready to engage in the full application process and about whether they meet the eligibility requirements before signing and submitting an initial application form. The goal of the planning tools is to help applicants who submit the initial application with completing steps of the application process.

To capture these goals, the primary outcomes will be the percentage of potential applicants (who have completed the online orientation) who sign the RFA initial application forms within the study’s 180-day follow-up period; have their case moved to “applying” status within 180 days; and complete the Phase I application steps within both 90 days and 180 days.

The interventions are hypothesized to impact the rates of all the outcomes, but only expected to impact the speed of completion of the Phase I application steps (with the planning tools being the driver of this hypothesized increase in speed). The 90-day outcome for completion of the Phase I application steps will allow us to capture this hypothesized increase in speed. In addition, DCFS would prefer applicants to complete all application steps within 90 days from signing the RFA initial application form, although they recognize that cases do not always meet that timeline. Analyzing this outcome at both 90 days and 180 days after completion of the online orientation will capture a range of cases that will likely be processed.

For the impact evaluation, potential community resource family applicants were randomly assigned to receive the readiness assessment (program group A), the readiness assessment and planning tools (program group B), or neither (control group). The point of random assignment was when potential applicants clicked on a link on the DCFS website to access the Binti application portal. Once an applicant creates a Binti account, the applicant will permanently be assigned to the flow they were randomly assigned to. Even though random assignment will be done at this early stage in the process, the study sample will be limited to those applicants who complete online orientation. The study sample will not include those who do not create an account, as well as those who create an account but do not complete orientation. The evaluation will compare outcomes of the control group to the program groups for the full sample. In addition to the impact study, BIAS-NG is conducting accompanying implementation and cost analyses to document how the intervention was delivered and at what cost.

External Link(s)

Registration Citation

Citation

Anzelone, Caitlin and Lawrence Katz. 2024. "Behavioral Interventions to Advance Self Sufficiency – Next Generation (BIAS-NG): Los Angeles County, Child Welfare." AEA RCT Registry. January 09. https://doi.org/10.1257/rct.12781-1.0

Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Experimental Details

Interventions

Intervention(s)

BIAS-NG aims to reduce the number of unqualified or unprepared families who apply to be community resource families and to increase the completion of application steps among families who do apply, thereby increasing community resource family approval rates using two interventions: a readiness check and planning tools. The readiness check is an online self-assessment to help potential applicants learn whether they are prepared to meet the requirements of the community resource family application process. It employs behavioral strategies, such as increasing salience about readiness criteria and presenting an active choice about whether to begin application. The planning tools are embedded in the online application and reminder emails and are designed to support applicants with completing application steps in a timely manner by using behavioral strategies, such as anchoring applicants to DCFS’s preferred timeline and using implementation prompts to keep track of their application progress.

Intervention (Hidden)

To begin the DCFS resource family application process, most potential community resource applicants create accounts in a case management and online application system, operated through Binti, a company that provides online foster care software for child welfare agencies. DCFS links to the Binti portal via their website, and for the study, a custom short code was embedded onto these links so that potential applicants who clicked on them were randomly assigned to one of three versions of the application experience. The control version was DCFS’s existing application flow. The two program group versions included (1) the readiness check; and (2) the readiness check and the planning tools.

In all three versions of the Binti portal, potential applicants create an account, answer a few demographic questions, and then complete an online orientation or indicate they have attended an in-person orientation. In the control version, potential applicants could then access the RFA initial application form. In the two program group versions, the readiness check was inserted in the portal between orientation and the RFA initial application form.

The readiness assessment is an interactive self-assessment that aims to help applicants determine if they are ready to apply to be a community resource family. It asks 14 questions organized into five categories: Upcoming Availability, Household Information, Financial Information, Caring for Children, and Legal History. The tool does not determine whether families may or may not apply; rather, it provides information to help them understand whether it is a good time for them to move forward in the full application process, given their circumstances at the point in time when applicants must make that choice. Those with responses that indicate eligibility are told they are “ready” and encouraged to continue; those whose responses indicating they may not be eligible will be encouraged to either “pause and consider” or “wait”, depending on the type of their readiness issue. These results are intended to anchor applicants on a course of action before they are presented with an option to continue to the initial application form or not. The tool also provides potential applicants with personalized action items to help them get ready and return to the process later, if appropriate. At the end of the readiness check, applicants can then proceed to the initial application form regardless of their results.

In all three groups, once families electronically “sign” (i.e., submit) the initial application, the Binti system alerts DCFS that a new family has completed an initial application form. At this point, a DCFS intake worker sends an email to the applicant to confirm that they want to move forward with the full application process. Upon receiving an affirmative response, an intake worker officially opens the case in the state’s Child Welfare Services Case Management System (CWS CMS), puts the case into the “applying” status in Binti, and assigns the family to an ORU caseworker. Applicants receive an email notifying them that they have been assigned to a caseworker.

After completing the initial application form, the control group and program group A (readiness check only) move to another page in Binti with the rest of the forms that must be completed during the application process. Program group B (readiness check and planning tools) has an intermediary step before accessing the rest of the application forms. This group views a “planning checklist” page in Binti that breaks the application process down into 26 steps. The checklist also tells applicants when they should complete each step and shares additional information about completing steps. Applicants can access a PDF version of the form they can print out and customize with the specific dates they will complete each step.

After reviewing the planning checklist, the applicants in program group B can access the application forms page. They have access to the same information as the other two groups, but they also have access to additional components of the planning tools. These include three additional reminder emails that are timed to remind applicants to complete key steps, more information about how to complete select application steps, and an updated flow on the Binti website that separates the forms for the home inspection and psych-social assessments onto a different page in Binti. The planning tools were designed to also include recommended deadlines to complete the application steps listed on the applications form page, but this component was not implemented as intended.

After completing the initial application form and accessing the rest of the application forms in Binti, all three groups must take a series of actions to trigger next steps in the full application process. All adults in an applicant’s household must attend an in-person appointment to provide fingerprints (“LiveScan”) so a DCFS intake worker may start the criminal background checks required by the county and state. Applicants must also upload additional forms (e.g., financial information, employment verification, health screen) and enroll for 12 hours of required resource family trainings. Once an applicant’s Binti account is complete with all required forms, background check results, and certificates of training completion (collectively “Phase I” application steps), the ORU supervisor reviews the case. Next, an RFA approval caseworker is assigned to take over the case to complete home assessments and a psycho-social assessment. Upon completion of both assessments, an applicant family may be approved or denied.

The intervention was first implemented on April 27, 2022. While monitoring sample build-up, the BIAS-NG team identified that some individuals were accessing the Binti portal through the DCFS website without going through random assignment. To address this issue, updates were made to links on the DCFS website to ensure that people accessing Binti through the website were randomly assigned as intended. Only individuals who created accounts on Binti after this update was made were included in the study sample, and therefore, the study period began on June 3, 2022. Applicants who created Binti accounts between June 3, 2022 and April 30, 2023, and who completed the online orientation in Binti were enrolled into the study sample.

Intervention Start Date

2022-04-27

Intervention End Date

2023-10-31

Primary Outcomes

Primary Outcomes (end points)

i. % of online orientation completers who signed the initial application forms within 180 days
ii. % of online orientation completers who had their case moved to “Applying” status within 180 days
iii. % of online orientation completers who completed the Phase I application steps within:
a. 90 days
b. 180 days

All outcomes are defined relative to the date of online orientation completion.

Primary Outcomes (explanation)

The primary outcomes will be created using administrative data provided by DCFS from Binti. For the measure about signing the initial application forms, the data provide the date the forms were signed. For the measure about “Applying” status, the data provide each status that a DCFS staff member assigned an application during the application process (e.g., recruiting, applying, withdrawn, approved) and the associated date. For the completion of the Phase I application steps measure, the data provide the names of caseworkers assigned to the application, and the date the caseworker was assigned. The completion of Phase I is indicated by a case being transferred from an ORU caseworker to an RFA caseworker.

Main Analysis

Effect of readiness assessment on signing initial application forms (Control group (C) vs Program group A (E1) + Program group B (E2)): As both treatment arms are identical up to the point of application, we plan to look at the impact on initial application signed by pooling both treatment arms as compared to the control group to increase power.

Effect of readiness assessment on all other outcomes (C vs E1): This comparison will assess whether the readiness assessment had an impact on the subsequent outcomes.

Effect of combined intervention on all other outcomes (C vs E2): This comparison will assess whether the readiness assessment and planning tool combined had an impact on the subsequent outcomes.

Effect of planning tool on all other outcomes (E1 vs E2): This comparison will assess the effect of adding the planning tool to the readiness assessment on the subsequent outcomes.

Secondary Outcomes

Secondary Outcomes (end points)

i. % of online orientation completers who completed the Phase I application steps within:
a. 30 days
b. 60 days
c. 120 days
ii. % of online orientation completers whose application was:
a. approved within 90 days
b. approved within 120 days
c. approved within 180 days
d. pending after 180 days

All outcomes are defined relative to the date of online orientation completion.

Secondary Outcomes (explanation)

If the intervention achieved impacts on any of the confirmatory outcomes, then we will conduct further analysis to examine whether the intervention had an impact on the secondary outcomes. If the intervention did not reduce application rates, reduce early dropout rates, and increase completion of Phase I steps, it will be unlikely to have an impact on the Resource Family approval rates.

The secondary outcomes will be created using administrative data provided by DCFS from Binti. For the completion of the Phase I application steps measure, the data provide the names of caseworkers assigned to the application, and the date the caseworker was assigned. The completion of Phase I is indicated by a case being transferred from an ORU caseworker to an RFA caseworker. For the outcomes related to approval, the data provide each status that a DCFS staff member assigned an application during the application process (e.g., recruiting, applying, withdrawn, approved) and the associated date. Approval status is indicated when the application is approved, and all other statuses indicate they were not approved, which can be because they are still applying after 180 days or because they have left the application process.

Exploratory Analysis

Effect of planning tool on all other outcomes among application signers (E1 signers vs E2 signers): In addition to the main experimental analysis among all online orientation completers, we will conduct an exploratory analysis of the remaining primary measures (and secondary measures if analyzed in the main analysis) among those who signed the initial application forms within 180 days. Because there shouldn’t be a differential effect on signing the initial applications forms between E1 and E2 groups, a comparison of the two intervention groups is valid to test statistically and will assess the effect of adding the planning tool to the readiness assessment on the subsequent outcomes.

Effect of readiness assessment on all other outcomes among application signers (C signers vs E1 signers): A comparison of the readiness assessment group to the control group will be non-experimental and will show the difference in rates on the subsequent outcomes.

Effect of combined intervention on all other outcomes among application signers (C signers vs E2 signers): A comparison of the combined readiness assessment and planning tool group to the control group will be non-experimental and will show the difference in rates on the subsequent outcomes.

Other Exploratory Analysis

The evaluation will investigate whether the planning tool worked differently for particular subgroups. Data is available from Binti for the subsample of people who sign and submit the initial application form, which is placed after the readiness check and before the planning tool. This data will be quality checked to determine its usability. Subgroups on prior experience with being a foster parent, annual income of applicant, and education level of applicant will be analyzed.

The data from Binti for the subsample of people who sign and submit the initial application form may also be used to identify characteristics of families that are more likely to drop out of the application process during the readiness assessment.

Experimental Design

The impact evaluation uses a randomized controlled trial design. Study participants should have had an equal (33/33/33) chance of being assigned to the control or two program groups. However, the final study sample has a ratio of (39/31/30).

The random assignment process was that a custom short code was inserted onto all links on the DCFS website that link to the Binti portal. The short code uses a function to shuffle an array, and three different links to Binti were shuffled. These links accessed Binti templates that included the application experience for the control group and each treatment arm. Once a potential applicant created their Binti account, they would always return to the same Binti template that they were randomly assigned to.

In addition to entering the Binti portal through DCFS, applicants could also apply directly through the Binti portal. Any applicants that did would bypass the random assignment process and proceed with the status quo application process. Such applicants cannot be differentiated from control group sample that entered through the DCFS website. Thus, the final control group sample likely includes some applicants who entered directly through Binti.

The study sample is limited to those applicants who complete online orientation. The study sample will not include those who do not create an account to access the Binti portal, as well as those who create an account but do not complete orientation. The study sample includes applicants to be resource families that come from the community, are not seeking to foster or adopt a relative or non-related extended family member, and do not have a case belonging to the Department of Probation.

Experimental Design Details

Randomization Method

Custom short code on links used to access the online application portal

Randomization Unit

Randomization occurred at the level of the account in the online application portal. Each individual applicant or set of two co-applicants that created an account in the portal was randomly assigned.

Was the treatment clustered?

Experiment Characteristics

Sample size: planned number of clusters

N/A

Sample size: planned number of observations

Intended target sample size: 1,680 applicants Realized sample size: 1,935 applicants

Sample size (or number of clusters) by treatment arms

Intended: 560 control, 560 treatment arm 1, and 560 treatment arm 2

Realized: 762 control, 596 treatment arm 1, and 577 treatment arm 2

Minimum detectable effect size for main outcomes (accounting for sample design and clustering)

The below estimates use a two-tailed test with a significance level of 10% and assume 80% power with a realized sample size of 1,935 individuals. The smallest true effect we could detect from the intervention is: An effect of 5.6 percentage points on the rate of individuals who signed the initial application forms. An effect of 6.8 percentage points from treatment arm 1 on the rate of individuals who had their case moved to “Applying” status within 180 days. An effect of 6.8 percentage points from treatment arm 2 on the rate of individuals who had their case moved to “Applying” status within 180 days. An effect of 5.7 percentage points from treatment arm 1 on the rate of individuals who completed the Phase I application steps within 90 days and 6.5 percentage points for those who completed the steps within 180 days. An effect of 5.8 percentage points from treatment arm 2 on the rate of individuals who completed the Phase I application steps within 90 days and 6.6 percentage points for those who completed the steps within 180 days.

Supporting Documents and Materials

IRB