Public Attitudes Towards AI in Children's Social Care: An Online Conjoint Experiment with British Adults

Conjoint experiment, so a table of two AI systems using the following.

Instructions:
Children's services often receive reports, for example, from schoolteachers, about a child who might be at risk of abuse or neglect. The local council wants to use a computer program that learns from past cases to help social workers. This program helps predict if the child might be harmed if no action is taken.

Now, imagine you are a council worker tasked with choosing which program to pick. On the following pages, please indicate which of the two programs shown you would personally prefer to see used by the council.

You will see seven pairs of programs, please choose one from each pair.

Background:
Before the introduction of the program, social workers made correct decisions about 80 children out of every 100 children referred to them.
They made two kinds of mistakes:

"False negatives": 10 children from every 100 referred children were at risk of harm, but were missed by social workers, and their cases were closed.
"False positives": 10 children from every 100 referred children were not at risk of harm, but were incorrectly progressed to further investigation.
Below are descriptions of two programs that could help social workers with screening referrals. Please read the descriptions and indicate which of the two programs shown you would personally prefer to see used by the council.

Attributes:
- Use of the AI system, "How the program is used by social workers in decisions made about children" (levels: The program makes a preliminary decision that the social worker must override to change; The program only makes a recommendation to the social worker)
- Measuring overall transparency, "Information about the program is:" (levels: Easy to understand and publicly available; Hard to understand without expert advice, but publicly available; Not publicly available)
- Measuring transparency in individual decisions "For each specific child, information about how the program made a prediction is:" (levels:
- How many fewer at-risk children had their cases incorrectly closed (levels: 0, 2, 4, 6, 8, 10) Note for context: background includes information that for social workers alone "10 children from every 100 referred children were at risk of harm, but were missed by social workers, and their cases were closed".
- How many fewer safe children had their cases incorrectly progressed (levels: 0, 2, 4, 6, 8, 10) Note for context: background includes information that for social workers alone "10 children from every 100 referred children were not at risk of harm, but were incorrectly progressed to further investigation".

2025-07-21

2025-08-31

Forced choice of an AI system from the two presented options in each task. In terms of the estimands, these will primarily be measured using the accuracy of willingness to pay in the primary analysis, but Average Marginal Component Effects (AMCEs) or choice probabilities will also be considered.

Willingness to pay in a mixed logit framework can be calculated by dividing the coefficients by a chosen scale or "price" parameter (e.g., false positive reduction) or through a model fit in willingness-to-pay space, as explained here: https://www.jstatsoft.org/article/view/v105i10.

This will be an online conjoint experiment, consisting of six tasks with two randomly assigned options (with a seventh task as the first task, flipped to test for consistency), randomly assigned separately across five sets of attribute levels described above under "Interventions". The order of the attribute levels will be randomised across participants, but the order will be consistent across their set of six tasks.

Randomised done via oTree.

Individual

No

N/A

2211 British online participants

Conjoint experiment, so randomisation is done uniformly for each level across each attribute, e.g. separately for each accuracy measure and each procedural safeguard attribute. This should result in a roughly equal number of tasks completed across the different attribute levels.

Given that the willingness-to-pay estimates will be produced from complex models in a novel context, it is quite difficult to compute an appropriate MDES here. If the AMCEs standard in political science were used, then using the cjpower application from Freitag and Schuessler (2020) based on their paper Schuessler and Freitag (2020), yields an MDES of 0.03 or assuming the conservative correction for intra-respondent reliability in Clayton et al. (2023, see Statistical Methods) 0.039, meaning a difference of 3 or 3.9 percentage points in the choice probability can be detected. While this is larger than the smallest statistically significant effect size in Horvath et al. (2023), this is a more high-stakes setting with, therefore, perhaps stronger preferences over attributes, and this MDES is still relatively low. To obtain an MDES of 0.02 would require 4,904 respondents, which is beyond the scope of the available budget.

Trial Protocol for Prospective Conjoint Experiment