Human Experts and Artificial Intelligence: The Value of Human Input in Diagnostic Imaging

Last registered on June 28, 2023

View Trial History

Pre-Trial

Trial Information

General Information

Title

Human Experts and Artificial Intelligence: The Value of Human Input in Diagnostic Imaging

RCT ID

AEARCTR-0008799

Initial registration date

January 13, 2022

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published

January 14, 2022, 1:56 PM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated

June 28, 2023, 10:43 AM EDT

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Country

Viet Nam

Region

Primary Investigator

Name

Alex Moehring

Affiliation

Purdue University

Contact Primary Investigator

Other Primary Investigator(s)

PI Name

Nikhil Agarwal

PI Affiliation

Massachusetts Institute of Technology

Contact Investigator

PI Name

Tobias Salz

PI Affiliation

Massachusetts Institute of Technology

Contact Investigator

PI Name

Pranav Rajpurkar

PI Affiliation

Harvard Medical School

Contact Investigator

Additional Trial Information

Status

Completed

Start date

2022-01-13

End date

2022-05-31

Keywords

Behavior, Labor

Additional Keywords

JEL code(s)

Secondary IDs

Prior work

This trial does not extend or rely on any prior RCTs.

Abstract

We plan to investigate how human experts combine their own information with AI predictions when making assessments and decisions in the radiology domain. See the attached pre-analysis plan for full details.

External Link(s)

Registration Citation

Citation

Agarwal, Nikhil et al. 2023. "Human Experts and Artificial Intelligence: The Value of Human Input in Diagnostic Imaging." AEA RCT Registry. June 28. https://doi.org/10.1257/rct.8799-1.1

Sponsors & Partners

Experimental Details

Interventions

Intervention(s)

See the attached pre-analysis plan for full details.

Intervention Start Date

2022-01-14

Intervention End Date

2022-05-01

Primary Outcomes

Primary Outcomes (end points)

To measure the quality of diagnostic assessments and decisions we will primarily focus on the following primary outcomes variables for each pathology group.
1. Error in probability assessment
2. Incorrect treatment/followup recommendation

The primary pathology groups we will consider are:
1. Pooled outcomes for all pathologies
2. Pooled outcomes for all AI assisted pathologies

Primary Outcomes (explanation)

See the attached pre-analysis plan for full details.

Secondary Outcomes

Secondary Outcomes (end points)

1. Time-taken and measures of effort exerted to parse the information in the X-ray and the clinical history, with and without AI
2. Heterogeneity of treatment effects by pathology prevalence and AI performance

Secondary Outcomes (explanation)

See the attached pre-analysis plan for full details.

Experimental Design

We ask radiologists to read chest x-rays, randomizing whether they have access to the patient's clinical history and an AI tool. See the attached pre-analysis plan for full details of the experiment and analysis plan.

Experimental Design Details

Randomization Method

The attached pre-analysis plan contains the full randomization details. The order in which radiologists go through the different treatments in each session will be randomized to account for order effects. For each radiologist, we will randomly sample 60 cases to be read under each experimental condition.2 We will then randomly select a sequence of images from the set of image sequences satisfying the following two criteria: (1) there are 15 cases in each treatment arm per round and (2) each image is read in all treatment arms across the rounds.

The advantage of the within-subject design is that we observe each radiologist making many decisions under each treatment arm, which makes it easier to detect effects as it can control for across-subject heterogeneity. In addition, this design facilitates estimation of an economic model of decision making described later on that is used to study automation bias or neglect.

Randomization Unit

Randomization occurs at the patient case level for each radiologist.

Was the treatment clustered?

Yes

Experiment Characteristics

Sample size: planned number of clusters

We expect between 30-35 radiologists to complete the experiment.

Sample size: planned number of observations

Each radiologist reads 60 cases under all four treatment arms. This will result in 8,400 total reads if 35 radiologists complete the experiment.

Sample size (or number of clusters) by treatment arms

Each radiologist will read 60 cases under each of the four treatment arms. This will result in 2,100 observations per arm if 35 radiologists complete the experiment.

Minimum detectable effect size for main outcomes (accounting for sample design and clustering)

See the attached pre-analysis plan for full details of the power calculations.

Supporting Documents and Materials

IRB