|
Field
Intervention (Public)
|
Before
Our intervention entails providing teams of child welfare caseworkers with a decision aide tool. This tool provides summary information to the team during team discussions about whether or not to investigate a referral that was called in to child protective services.
The primary decision aide arm involves showing a 1-20 score (grouped into ventiles, where a 20 represents the highest 5% of risk of child home removal within two years). The tool uses machine learning methods trained using several years of historic administrative data and is highly predictive future home removal (e.g., of any child to foster care), re-referral to child protective services (CPS), and child maltreatment death. The tool makes use of statewide child welfare data (e.g., past referrals) and public benefits data (e.g., SNAP, benefit denial). Team members are provided with a pdf (digital or hard copy) of a figure showing the relationship between risk score and historic removal rate for context. The risk tool consists of a back-end database and a front-end user interface that can be accessed by a team as they make a decision.
A team member will read the details of an incoming referral and procure certain family history items on record (e.g., past incidents), and a team member will write/type out those details in a document for the whole team to see. Our intervention requires a team member to check the score listed in the system interface and write the risk score in the case document for the team to see, and so we can confirm a team saw the score. A team will then discuss the case given the information on hand, and decide whether or not to send a worker to do nothing, provide services without investigation, or investigate (screen-in; and if so, assign an urgency level to the investigation). Intervention cases will require teams to check the decision aide tool, non-intervention cases will include the same discussion process except without decision tool information being available for the decision. The tool uses only past information (nothing from the present referral) and all of this information is in principle available for a team without the intervention should they choose to spend time exploring a child’s/family’s history. The tool also includes a non-score component that indicates which features contributed to the score being high or low. Our intervention will include multiple treatment arms with different types of information included with the decision tool.
|
After
Our intervention entails providing teams of child welfare caseworkers with a decision aide tool. This tool provides summary information to the team during team discussions about whether or not to investigate a referral that was called in to child protective services.
The primary decision aide arm involves showing a 1-20 score (grouped into ventiles, where a 20 represents the highest 5% of risk of child home removal within two years). The tool uses machine learning methods trained using several years of historic administrative data and is highly predictive future home removal (e.g., of any child to foster care), re-referral to child protective services (CPS), and child maltreatment death. The tool makes use of statewide child welfare data (e.g., past referrals) and public benefits data (e.g., SNAP, benefit denial). Team members are provided with a pdf (digital or hard copy) of a figure showing the relationship between risk score and historic removal rate for context. The risk tool consists of a back-end database and a front-end user interface that can be accessed by a team as they make a decision.
A team member will read the details of an incoming referral and procure certain family history items on record (e.g., past incidents), and a team member will write/type out those details in a document for the whole team to see. Our intervention requires a team member to check the score listed in the system interface and write the risk score in the case document for the team to see, and so we can confirm a team saw the score. A team will then discuss the case given the information on hand, and decide whether or not to send a worker to do nothing, provide services without investigation, or investigate (screen-in; and if so, assign an urgency level to the investigation). Intervention cases will require teams to check the decision aide tool, non-intervention cases will include the same discussion process except without decision tool information being available for the decision. The tool uses only past information (nothing from the present referral) and all of this information is in principle available for a team without the intervention should they choose to spend time exploring a child’s/family’s history. The tool also includes a non-score component that indicates which features contributed to the score being high or low. Our intervention will include multiple treatment arms with different types of information included with the decision tool.
*** 09/13/2022: Due to feasibility reasons from the partner organization, the trial only included a score treatment arm and not an explanation treatment arm.
|
|
Field
Primary Outcomes (End Points)
|
Before
Our primary outcomes are as follows. Outcomes will be measured as (1) difference between treatment and control, and (2) difference conditional on child’s underlying risk score. Analysis will be conducted at the referral, family, and child level.
Decision-Making:
- Number of total characters, discussion characters, and child welfare history characters typed during team decision process. These character counts will help to measure the effect of the tool on the attention given to different aspects of a referral and the referral as a whole, and potential effort spillovers to control cases.
- Time per team decision. Constructed using timestamps between cases, where the data are reliable.
- Rate of screen-in (investigation)
- Rate of screen-in by type: immediate, 3-day, 5-day, HRA, FAR (intensive margin of investigations)
- Rate of services provided/recommended
- Number of days until case closed
Investigations:
- Fraction of families found
- Fraction of investigations substantiated for abuse or neglect
- Fraction of investigations provided services within 1, 2, 3, 6, 12, and 24 months
- Fraction of investigations removed within 1, 2, 3, 6, 12, and 24 months
- Fraction of FAR changed to HRA (measure of how accurately case was originally classified)
- Rate of investigation in families where no services are open
False negatives:
- Fraction of screened-out cases re-referred within 1, 2, 3, 6, 12, and 24 months
- Fraction of screened out cases re-referred within 1, 2, 3, 6, 12, and 24 months with egregious injury (near fatal, death, either)
Overall:
-Fraction of total cases re-referred (or removed from home) within 1, 2, 3, 6, 12, and 24 months
-Fraction of total cases re-referred (or removed from home) within 1, 2, 3, 6, 12, and 24 months with egregious injury
- Fraction of investigations provided services within 1, 2, 3, 6, 12, and 24 months
-Fraction of cases receiving family visitor or CCR
- Fraction of total referrals with hospital or Medicaid claim for broken bone, ED visit, avoidable injury, unavoidable injury, any injury, well-child visit, vaccination, and asthma ED visit within 1, 2, 3, 6, 12, 24, and 36 months
- Child truancy and standardized test scores within 1, 2, 3, 6, 12, 24, and 36 months
|
After
Our primary outcomes are as follows. Outcomes will be measured as (1) difference between treatment and control, and (2) difference conditional on child’s underlying risk score. Analysis will be conducted at the referral, family, and child level.
Decision-Making:
- Number of total characters, discussion characters, and child welfare history characters typed during team decision process. These character counts will help to measure the effect of the tool on the attention given to different aspects of a referral and the referral as a whole, and potential effort spillovers to control cases.
- Time per team decision. Constructed using timestamps between cases, where the data are reliable.
- Rate of screen-in (investigation)
- Rate of screen-in by type: immediate, 3-day, 5-day, HRA, FAR (intensive margin of investigations)
- Rate of services provided/recommended
- Number of days until case closed
Investigations:
- Fraction of families found
- Fraction of investigations substantiated for abuse or neglect
- Fraction of investigations provided services within 1, 2, 3, 6, 12, and 24 months
- Fraction of investigations removed within 1, 2, 3, 6, 12, and 24 months
- Fraction of FAR changed to HRA (measure of how accurately case was originally classified)
- Rate of investigation in families where no services are open
False negatives:
- Fraction of screened-out cases re-referred within 1, 2, 3, 6, 12, and 24 months
- Fraction of screened out cases re-referred within 1, 2, 3, 6, 12, and 24 months with egregious injury (near fatal, death, either)
Overall:
-Fraction of total cases re-referred (or removed from home) within 1, 2, 3, 6, 12, and 24 months
-Fraction of total cases re-referred (or removed from home) within 1, 2, 3, 6, 12, and 24 months with egregious injury
- Fraction of investigations provided services within 1, 2, 3, 6, 12, and 24 months
-Fraction of cases receiving family visitor or CCR
- Fraction of total referrals with hospital or Medicaid claim for broken bone, ED visit, avoidable injury, unavoidable injury, any injury, well-child visit, vaccination, and asthma ED visit within 1, 2, 3, 6, 12, 24, and 36 months
- Child truancy and standardized test scores within 1, 2, 3, 6, 12, 24, and 36 months
*** 09/22/2022: We are preparing to receive linked hospital inpatient and ED records, as well as detailed text data from discussions, and wanted to pre-specify our main analyses for these records in advance as we know have just gotten access to the codebooks. All text below was added on 09/22/2022.
For hospital outcomes, to clarify what we had previously specified, we plan to examine both the extensive and intensive margins of child hospital visits (any instance and number of instances for each child and for all children in the household, as well as being first ICD code listed if applicable) over time of:
• ED visits
• Admissions of priority type: Emergency, Urgent, and Trauma
• Visits listing any injury, any intentional and any unintentional injuries
• Preventive medicine codes including well-child visit and vaccination (if not too few outpatient records)
• Preventable (vs unpreventable) child ED visits, using the Ambulatory care-sensitive conditions (ACSC) which are conditions that could be managed or addressed well in outpatient settings if attended to (such as asthma, dehydration, anaemia, etc)
• ICD-10 codes that are both suspected and suggestive of child maltreatment by the existing literature. ICD-10-CM codes are for suspected and confirmed maltreatment and so have been shown to underestimate instances of child maltreatment (Hughes et al. 2021), so we will look at a broader set of ICD codes suggestive of maltreatment. ICD-10 codes that are considered predictive of future mortality using the trauma mortality prediction model (TMPM). We plan to examine effects for all kids and kids 11-17years old as – to the best of our current knowledge – the TMPM has only been validated for kids 11-17years (Cassidy et al. 2014).
• ICD-10 codes for child cancer that are considered placebos (where we do not expect to find an effect).
• ICD-10 codes indicating exposure to substances (e.g., drugs)
• Cost of medical care provided paid for by public insurers, by non-insured clients and gone unpaid
We also plan to receive de-identified notes from the team discussions. We hope to use natural language text processing machine learning methods to explore rigorously what might be changing in the discussions. Of particular interest to us is to assess whether the tool changes workers’ attention and sensitivity to the severity of the current allegation (such as words suggesting or indicating a potential child injury).
Contrary to what we had originally hoped and listed in the original pre-register, accessing school records will not be possible.
|
|
Field
Intervention (Hidden)
|
Before
Our intervention will include multiple treatment arms with different information features by the decision tool. These include (at the mother level for each referral):
1. No decision tool support (control)
2. Risk score and explanation
3. Risk score only
4. Explanation only
The primary intervention consists of a risk score (1-20 integer) and explanation features. These features include the relative importance of certain risk factors in contributing to the quantitative score (roughly the weights assigned by the machine learning algorithm; flagging elements that led were risk-inducing or mitigating factors) and a sandbox feature where team members can change feature values to see how the predictive score would respond.
The risk only intervention consists only of a risk score without any explanation features. The explanation only intervention consists of the risk factor contributions but omits the actual integer score.
For a given referral, all children on the referral receive an individual risk score that is observable to the team.
Foremost, from our intervention we hope to learn whether the complete tool (score + explanation) leads to changes in decision-making and child outcomes. Currently, many of the lowest risk cases get screened in for investigation and many of the highest risk cases get screened out, suggesting possible efficiency gains from using the tool. It would be interesting to see if teams are influenced by the tool, and whether relevant outcomes also improve.
In addition to evaluating the full tool, I also hope to learn (1) what information most affects decisions, and (2) how? In particular, we plan to run two supplementary treatment arms of score only (i.e., team given the "utility weights" but not the information) and explanation/feature flag only (team given information but not given utility weights). Although these arms will likely be underpowered, they are both policy- and potentially theory-relevant, and we hope they can provide suggestive evidence for future work. To understand mechanisms, we plan to estimate a simple control group or pre-intervention model of the implicit weights workers place on case features when making a decision, and how those weights change when given access to the tool.
|
After
Our intervention will include multiple treatment arms with different information features by the decision tool. These include (at the mother level for each referral):
1. No decision tool support (control)
2. Risk score and explanation
3. Risk score only
4. Explanation only
The primary intervention consists of a risk score (1-20 integer) and explanation features. These features include the relative importance of certain risk factors in contributing to the quantitative score (roughly the weights assigned by the machine learning algorithm; flagging elements that led were risk-inducing or mitigating factors) and a sandbox feature where team members can change feature values to see how the predictive score would respond.
The risk only intervention consists only of a risk score without any explanation features. The explanation only intervention consists of the risk factor contributions but omits the actual integer score.
For a given referral, all children on the referral receive an individual risk score that is observable to the team.
Foremost, from our intervention we hope to learn whether the complete tool (score + explanation) leads to changes in decision-making and child outcomes. Currently, many of the lowest risk cases get screened in for investigation and many of the highest risk cases get screened out, suggesting possible efficiency gains from using the tool. It would be interesting to see if teams are influenced by the tool, and whether relevant outcomes also improve.
In addition to evaluating the full tool, I also hope to learn (1) what information most affects decisions, and (2) how? In particular, we plan to run two supplementary treatment arms of score only (i.e., team given the "utility weights" but not the information) and explanation/feature flag only (team given information but not given utility weights). Although these arms will likely be underpowered, they are both policy- and potentially theory-relevant, and we hope they can provide suggestive evidence for future work. To understand mechanisms, we plan to estimate a simple control group or pre-intervention model of the implicit weights workers place on case features when making a decision, and how those weights change when given access to the tool.
*** 09/13/2022: Our trial design remained largely unchanged, but due to feasibility constraints our implementing partner asked that we focus on two treatment arms: (1) score shown and (2) no score shown (control). We do not provide a text explanation for the score in either treatment arm.
|