A Sufficient Statistic for Designing and Evaluating Human-AI Collaboration

Last registered on June 17, 2025
View Trial History

Pre-Trial

Trial Information

General Information

Title
A Sufficient Statistic for Designing and Evaluating Human-AI Collaboration
RCT ID
AEARCTR-0013990
Initial registration date
July 10, 2024

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
July 15, 2024, 10:12 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
June 17, 2025, 11:23 AM EDT

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Country
United States of America
Region

Primary Investigator

Name
Alex Moehring
Affiliation
Purdue University
Contact Primary Investigator

Other Primary Investigator(s)

PI Name
Nikhil Agarwal
PI Affiliation
MIT
Contact Investigator
PI Name
Alexander Wolitzky
PI Affiliation
MIT
Contact Investigator

Additional Trial Information

Status
Completed
Start date
2024-07-10
End date
2024-09-30
Keywords
Behavior, Labor
Additional Keywords
Artificial Intelligence, Human-AI Interaction
JEL code(s)
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
We cannot risk that participants become aware of this information before the study is complete. Please refer to the pre-analysis plan for more information.
External Link(s)

Registration Citation

Citation
Agarwal, Nikhil, Alex Moehring and Alexander Wolitzky. 2025. "A Sufficient Statistic for Designing and Evaluating Human-AI Collaboration." AEA RCT Registry. June 17. https://doi.org/10.1257/rct.13990-1.2
Sponsors & Partners
Experimental Details

Interventions

Intervention(s)
We cannot risk that participants become aware of this information before the study is complete. Please refer to the pre-analysis plan for more information.
Intervention (Hidden)
We study a new sufficient statistic approach to designing the optimal collaboration between humans and AI algorithms in the context of fact-checking. In this approach, we first estimate human accuracy with access to AI assistance and estimate how this accuracy varies by the AI signal provided. We then use these estimates to calculate optimal disclosure and automation policies which we test experimentally in a second round. The attached pre-analysis plan contains the full details.
Intervention Start Date
2024-07-10
Intervention End Date
2024-09-30

Primary Outcomes

Primary Outcomes (end points)
We cannot risk that participants become aware of this information before the study is complete. Please refer to the pre-analysis plan for more information.
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
We cannot risk that participants become aware of this information before the study is complete. Please refer to the pre-analysis plan for more information.
Experimental Design Details
We study a new sufficient statistic approach to designing the optimal collaboration between humans and AI algorithms in the context of fact-checking. In this approach, we first estimate human accuracy with access to AI assistance and estimate how this accuracy varies by the AI signal provided. We then use these estimates to calculate optimal disclosure and automation policies which we test experimentally in a second round. The attached pre-analysis plan contains the full details.
Randomization Method
We cannot risk that participants become aware of this information before the study is complete. Please refer to the pre-analysis plan for more information.
Randomization Unit
We cannot risk that participants become aware of this information before the study is complete. Please refer to the pre-analysis plan for more information.
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
We cannot risk that participants become aware of this information before the study is complete. Please refer to the pre-analysis plan for more information.
Sample size: planned number of observations
We cannot risk that participants become aware of this information before the study is complete. Please refer to the pre-analysis plan for more information.
Sample size (or number of clusters) by treatment arms
We cannot risk that participants become aware of this information before the study is complete. Please refer to the pre-analysis plan for more information.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Supporting Documents and Materials
IRB

Institutional Review Boards (IRBs)

IRB Name
MIT Committee on the Use of Humans as Experimental Subjects
IRB Approval Date
2024-01-18
IRB Approval Number
E-5579
IRB Name
Purdue University Human Research Protection Program
IRB Approval Date
2024-07-09
IRB Approval Number
2024-1030
Analysis Plan

Analysis Plan Documents

Pre-Analysis Plan

MD5: bc8293bf9635b2ea38a2c69a9812f700

SHA1: 3a68f5edf092455fc4480a0c7e06c6b82dd53aa0

Uploaded At: July 12, 2024

Amendment to Pre-Analysis Plan for Round 2

MD5: 355056bc716befb9d73593721963e70f

SHA1: 85103077c2e5a469b9d8b3746a4dadba6707905a

Uploaded At: August 30, 2024

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials