Coproduction for algorithm adherence in AutoML applications

Last registered on November 30, 2022
View Trial History

Pre-Trial

Trial Information

General Information

Title
Coproduction for algorithm adherence in AutoML applications
RCT ID
AEARCTR-0010449
Initial registration date
November 26, 2022

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
November 30, 2022, 4:21 PM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Country
Germany
Region

Primary Investigator

Name
Marius Protte
Affiliation
Universität Paderborn
Contact Primary Investigator

Other Primary Investigator(s)

Additional Trial Information

Status
In development
Start date
2022-11-28
End date
2022-12-21
Keywords
Behavior, Firms & Productivity
Additional Keywords
JEL code(s)
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
AutoML is a promising field of Machine Learning in which even non-experts can participate in the advantages of data science and, for instance, benefit from predictions of algorithms. However, non-experts cannot necessarily evaluate the benefits of AutoML predictions and, hence, decide against those predictions and rather trust in their own expertise of their field even if it is not rational to do so (i.e., algorithm aversion; Burton et al. 2020, Jussupow et al. 2020). For instance, an engineer who is responsible for the maintenance of a production unit and worked at the machine for several years, determines the timing of maintenance based on his experience no matter what a recently installed maintenance artificial intelligence (AI) predicts. As AI is more efficient for many tasks, we investigate how to foster trust of domain experts in AI technologies. Coproduction (i.e., the engagement of customers in the production process) has shown the strived effects in the provision of services as well as classic production set-ups. There is also evidence, that algorithm aversion decreases when users can modify these algorithms (Dietvorst et al. 2018)
In our experimental study we examine whether this effect holds in an AutoML set-up. We create an environment, in which subjects act as engineers and are confronted with an AI that helps to predict malfunctions of their machine. We implement two groups (i.e., baseline group and coproduction group) and compare between those groups whether coproduction indeed fosters trust of the subjects resulting in decisions that are more conform with the AI predictions.
External Link(s)

Registration Citation

Citation
Protte, Marius. 2022. "Coproduction for algorithm adherence in AutoML applications." AEA RCT Registry. November 30. https://doi.org/10.1257/rct.10449-1.0
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
2 group between-subject design: coproduction treatment (active training of AI) vs. control (passive witnessing of AI training results) --> see experimental design

Hypothesis 1: We expect individuals who actively contributed to the AI’s training to be more likely to adhere to the AI’s advice compared to individuals who were not actively involved in the algorithm’s training.

Hypothesis 2: We expect that for those individuals whose assessment of the situation differs from that of the AI, those individuals are more likely to follow the AI's advice if their initial assessment differs fundamentally from the AI's advice.
Intervention Start Date
2022-11-28
Intervention End Date
2022-12-06

Primary Outcomes

Primary Outcomes (end points)
Adherence to algorithmic advice
Primary Outcomes (explanation)
Fraction of decisions in which an individual complies with the AI's advice

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
In our experiment, participants play in the role of an engineer that is responsible for assessing the likelihood of a production machine malfunctioning and deciding whether machine maintenance should take place in a given situation.

The experiment consists of four stages. In the first, participants read the experimental instructions and subsequently answer extensive comprehension checks to ensure a sufficient understanding of the experimental design and task rules. In the second stage, participants set their personal acceptance ranges for three machine parameters (temperature, speed and voltage) for which they deem operating the machine innocuous. In the third stage (“training stage”), the training of the AI takes place. In the control group, this training happens automatically with no decision-making authority to the participants, who are simply informed about the training’s success, i.e., the AI’s precision, in the end. In the treatment group, participants actively evaluate the training data themselves and thereby contribute to training the AI and consequently its precision. In stage four, the participants supervise the machine for 25 rounds. In each round, they decide whether machine maintenance should or should not take place. In these maintenance decisions, participants receive recommendations from the AI on the likelihood of a machine malfunction. Participants give an initial assessment of the machine status and their maintenance decision first and are then able to revise their decision after learning about the AI’s advice.

Participants receive performance-dependent monetary incentives for their decisions plus a fixed show-up fee.
Experimental Design Details
Randomization Method
Randomization done by experiment recruiting software ORSEE from a pool of about 2,500 students of Paderborn University
Randomization Unit
Individual
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
120 student participants
Sample size: planned number of observations
120 student participants
Sample size (or number of clusters) by treatment arms
60 participants treatment group (active coproduction in algorithm training), 60 participants control group (passive algorithm training)
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Supporting Documents and Materials
IRB

Institutional Review Boards (IRBs)

IRB Name
IRB Approval Date
IRB Approval Number
Analysis Plan

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials