Algorithmic RCT in Child Protection Services

Last registered on December 20, 2023
View Trial History

Pre-Trial

Trial Information

General Information

Title
Algorithmic RCT in Child Protection Services
RCT ID
AEARCTR-0012623
Initial registration date
December 11, 2023

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
December 20, 2023, 9:43 AM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Primary Investigator

Name
Arkadev Ghosh
Affiliation
Duke University
Contact Primary Investigator

Other Primary Investigator(s)

PI Name
Jason Baron
PI Affiliation
Duke University
Contact Investigator
PI Name
Jeongsoo Suh
PI Affiliation
Duke University
Contact Investigator

Additional Trial Information

Status
In development
Start date
2023-12-13
End date
2025-06-01
Keywords
Behavior, Labor
Additional Keywords
JEL code(s)
I38
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
Every year, around 4 million referrals are made to Child Protective Services in the United States, involving more than 7 million children. Approximately half of these reports are screened in for investigation. Screened-in referrals are assessed by a caseworker, with support from their supervisor. There are few tools that are available to supervisors to support their practice and encourage rich conversations with caseworkers. A county in the U.S. is planning to launch the CSDA Supervision Tool, a data lead tool developed by professors at UNC-Chapel Hill and Auckland University of Technology and Centre for Social Data Analytics (CSDA). This study will conduct an impact evaluation to assess whether these tools are (i) improving outcomes for families and (ii) not having any adverse outcomes. A key feature of the study will be its focus on treatment heterogeneity by workers' baseline beliefs about the risk distribution of different demographic groups.
External Link(s)

Registration Citation

Citation
Baron, Jason, Arkadev Ghosh and Jeongsoo Suh. 2023. "Algorithmic RCT in Child Protection Services." AEA RCT Registry. December 20. https://doi.org/10.1257/rct.12623-1.0
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
The CPS agency is launching the CSDA Supervision Tool, a data lead tool developed by professors at UNC-Chapel Hill and Auckland University of Technology and Centre for Social Data Analytics (CSDA). The goal of the tool is to optimally target agency resources to child maltreatment investigations based on their risk profile.
Intervention Start Date
2023-12-18
Intervention End Date
2025-01-01

Primary Outcomes

Primary Outcomes (end points)
Measure of safety using CPS data: re-referrals to the agency, subsequent substantiated investigations, subsequent foster care placements, racial disparities in safety and agency responses. For each of these outcomes, we will get the data directly from the CPS agency. We will also conduct a survey at multiple points of the RCT, in which we will assess workers' beliefs regarding the risk profile of different demographic groups (by race and by gender)---and how these beliefs may be changing as a result of the intervention.
Primary Outcomes (explanation)
(1) Re-referrals to the agency: a subsequent report or investigation after the conclusion of the focal investigation
(2) Subsequent substantiated investigations: same as (1), but where the report/investigation is substantiated
(3) Subsequent foster care placement: same as (1), but where the report/investigation culminates in foster care placement
(4) Racial disparities in safety: (1) - (3) but with a focus on heterogeneity by race
(5) Racial disparities in agency responses to current investigations: racial differences in foster care placement rates or service provision rates following an investigation
(6) Belief updating: we will have access to a baseline belief survey, in which workers were asked to report their beliefs regarding the risk profile of different demographic groups (race and gender). We will conduct additional surveys during the intervention to analyze belief updating.

Secondary Outcomes

Secondary Outcomes (end points)
(1) Variance of service provision across investigators. We also *hope* to get external data measuring safety such as hospitalization records. (2) Heterogeneous treatment effects by investigators' characteristics, including observable traits -- particularly, gender, family characteristics, and baseline trust in the algorithm. (3) "First stage" outcomes such as provision of prevention services, time spent on investigations, face-to-face interaction with the family's home, and effort during investigations.
Secondary Outcomes (explanation)
(1) Variance of service provision across investigators: Does the algorithm lead to more consistent (lower variance) in service/foster care placement rates across investigators within the agency, conditional on case characteristics?
(2) Hospitalization records: hospital visits with an injury ICD code that are possibly due to child maltreatment (based on availability)
(3) It would be important to understand how the intervention affected investigator effort and provision of services that may in turn affect downstream child outcomes. We will consider measures such time spent on investigation, number of face-to-face interactions with the family's home and "effort' exerted by investigators, by looking at length and quality of their notes.

Experimental Design

Experimental Design
The RCT is designed to estimate the causal effects of the supervision tool on the agency's decisions and children's subsequent outcomes. Randomization will take place at the household level. The first time the agency receives a report for a given family, the supervisor will be randomly shown the family's risk score or not. The supervisor will then communicate this score to the investigator/caseworker, who will consider the score when deciding the appropriate course of action during the investigation/assessment. Families will keep their initial randomization status throughout the trial period.
Experimental Design Details
Not available
Randomization Method
Randomization is done in a computer at the agency's office.
Randomization Unit
Household
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
This will depend on how many cases arrive in the following year. Our best guess is that, in this county, we will see about 1000-3000 households during our trial period.
Sample size: planned number of observations
We estimate that any given household will have an average of two children. Any given report may contain information on other children in the household, besides the alleged victim. Therefore, we expect roughly 2000-6000 observations.
Sample size (or number of clusters) by treatment arms
Roughly 500-1500 clusters in each group (treatment versus control).
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Supporting Documents and Materials
IRB

Institutional Review Boards (IRBs)

IRB Name
Duke University Campus IRB
IRB Approval Date
2023-02-07
IRB Approval Number
2023-0198
Analysis Plan