An Experimental Study of AI Decision Support and Human Judgment

Last registered on June 29, 2026
View Trial History

Pre-Trial

Trial Information

General Information

Title
An Experimental Study of AI Decision Support and Human Judgment
RCT ID
AEARCTR-0019001
Initial registration date
June 23, 2026

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
June 29, 2026, 8:42 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Primary Investigator

Name
Kjetil Veen Rønning
Affiliation
University of Stavanger
Contact Primary Investigator

Other Primary Investigator(s)

Additional Trial Information

Status
In development
Start date
2026-06-26
End date
2026-11-02
Keywords
Behavior, Firms & Productivity, Labor
Additional Keywords
automation bias, confirmation bias, trust calibration, human–ai interaction, ai decision support, overreliance, generative ai, decision-making, cognitive bias
JEL code(s)
D91, C91, D83
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
Artificial intelligence (AI) is increasingly deployed as decision support in organisations. This study examines how prior exposure to AI advice affects subsequent human judgment in case-based decision tasks.

The study is an online experiment with 1:1 individual randomisation. Participants complete three short decision cases involving managerial judgment under uncertainty. The target sample is at least 200 eligible participants with formal leadership responsibility, defined as at least one year of formal responsibility for leading, coordinating, or professionally directing the work of others.

In the treatment condition, participants complete all three cases in a sequential two-step format: they first provide an initial answer without AI support, then receive AI advice and are given the opportunity to revise their answer. In the control condition, participants complete Case 1 and Case 2 without AI support only. In Case 3, both groups follow the same two-step procedure by first providing an initial answer, then receiving AI advice and being given the opportunity to revise.

The primary outcome is an ordinal measure of whether, and how strongly, participants revise their Case 3 answer toward, or away from, the AI recommendation after seeing it. The primary test compares this revision between treatment and control using ordinal logistic regression on the full eligible sample, without conditioning on the initial answer. Secondary analyses examine decision quality in Cases 1 and 2, confidence change, perceived mental effort, perceived task difficulty, and time-on-task. All AI messages are pre-written and fixed across participants. No live language model is used.
External Link(s)

Registration Citation

Citation
Rønning, Kjetil Veen. 2026. "An Experimental Study of AI Decision Support and Human Judgment." AEA RCT Registry. June 29. https://doi.org/10.1257/rct.19001-1.0
Sponsors & Partners
Experimental Details

Interventions

Intervention(s)
The intervention consists of pre-written AI decision-support messages embedded in an online decision-making experiment with participants who have formal leadership responsibility. Leadership responsibility is self-reported. Participants in the treatment condition receive AI advice after their initial answer in each of the three cases and may revise their answer. Participants in the control condition do not receive AI advice in Cases 1 and 2, but receive the same AI-advice procedure as treatment participants in Case 3. The purpose is to examine how prior exposure to AI decision support affects later reliance on AI advice.
Intervention Start Date
2026-06-26
Intervention End Date
2026-08-15

Primary Outcomes

Primary Outcomes (end points)
The primary outcome is an ordinal measure (move_c) of whether, and how strongly, participants revise their Case 3 answer toward, or away from, the AI recommendation after seeing it. The primary test compares this revision between treatment and control using ordinal logistic regression on the full eligible sample, with treatment arm as predictor. A pre-specified balance check on the initial-answer distribution gates an initial-answer-adjusted model, which is reported as a robustness analysis.
Primary Outcomes (explanation)
Case 3 has three ordered answer options, coded A = 0, B = 1, and C = 2, where Option C is the AI-recommended option. The primary outcome move_c is the signed change from the initial to the post-advice answer, oriented so that positive values indicate movement toward the AI recommendation and negative values movement away from it. As a signed change score it is defined for every participant and captures movement in both directions, including for those whose initial answer was already C.

The primary test compares move_c between treatment and control using ordinal logistic regression on the full eligible sample, with treatment arm as the predictor. This is an intention-to-treat estimate. No participant is excluded by their initial answer, and the initial answer is not used as a covariate, because it is measured after randomisation and may itself be affected by prior AI exposure. Effect sizes and confidence intervals are reported alongside significance.

Participants who start closer to Option C have less room to move toward it, so a pre-specified balance check compares the distribution of initial answers between arms. This check guides how the primary estimate is read, but does not change it. Two additional models are also reported, one adding the initial answer and one adding the pre-registered baseline covariates (AI experience, AI ability, AI decision use, AI trust, leadership years). If the proportional-odds assumption does not hold, a generalised ordered logit model is reported as a sensitivity analysis.

Secondary Outcomes

Secondary Outcomes (end points)
• Performance in Cases 1 and 2: absolute deviation between the participant’s answer and a pre-specified normative benchmark in each case. For the treatment group, performance is based on the post-AI answer. For the control group, performance is based on the single unaided answer.
• Within-treatment improvement in Cases 1 and 2: paired change in absolute deviation from pre-AI to post-AI answer.
• Adoption of the Case 3 AI recommendation: a binary outcome coded 1 if the participant's post-advice answer is Option C and 0 otherwise, computed over the full eligible sample.
• Confidence change: confidence ratings on a 1–7 scale collected before and after each case; confidence change is defined as post minus pre.
• Subjective mental effort: mental effort rating on a 1–7 scale after each case.
• Perceived task difficulty: perceived difficulty rating on a 1–7 scale after each case.
• Time-on-task: time spent on each case.
• Case 3 manipulation check: correct recall of the Case 3 AI recommendation.
• AI advice reading time in Case 3.
Secondary Outcomes (explanation)
Performance in Cases 1 and 2 is analysed using within-treatment pre/post comparisons and between-arm comparisons of absolute deviation from the normative benchmark. Holm–Bonferroni correction is applied across the two case-level performance tests.

The binary Case 3 outcome measures adoption of the Case 3 AI recommendation, that is, whether the participant ends on Option C. It is analysed as a between-arm intention-to-treat contrast on the full eligible sample, without conditioning on the initial answer. As a pre-specified sensitivity analysis, both the primary and secondary Case 3 analyses are repeated among participants who correctly recall the Case 3 AI recommendation.

Exploratory analyses examine whether patterns in the treatment effect are consistent with mediation by cumulative mental effort, cumulative perceived difficulty, and cumulative time-on-task across Cases 1 and 2. Additional exploratory moderation analyses examine baseline AI trust, AI usage frequency, self-rated AI ability, experience with AI as decision support, and leadership experience. These background variables are treated as covariates or moderators, not as outcomes. Holm–Bonferroni correction is applied across the five interaction tests.

Experimental Design

Experimental Design
Participants are recruited online through social media (e.g. LinkedIn), professional networks, and snowball distribution. Participants complete the experiment in a single online session. After consent and background questions, participants are randomised 1:1 to Treatment or Control at the start of the session.
All participants complete three short decision cases in a fixed order. The cases involve managerial judgment under uncertainty. In the treatment condition, participants provide an initial answer, report confidence, receive AI advice, and may revise their answer in each case. In the control condition, participants complete Cases 1 and 2 without AI support, and then follow the same two-step AI-advice procedure as treatment participants in Case 3.
The primary target sample is at least 200 eligible participants with formal leadership responsibility. Participants without leadership experience may complete the experiment but will be treated as a secondary convenience sample and are not included in the primary confirmatory analyses.
All AI messages are pre-written and fixed across participants within each condition. Participation is anonymous and voluntary; no personally identifying information is collected.
Experimental Design Details
Not available
Randomization Method
Randomisation is performed by the web-based experimental instrument at the start of each session. Allocation ratio is 1:1. Assignment is concealed from participants.
Randomization Unit
Individual
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
Not applicable
Sample size: planned number of observations
200 individuals with formal leadership responsibility, defined as at least one year of formal responsibility for leading, coordinating, or professionally directing the work of others. Recruitment will target this primary sample and data collection will continue until the pre-specified data collection end date. Participants without leadership experience may complete the experiment and may be retained as a secondary convenience sample, but they will not count toward the primary target sample and will not be included in the primary confirmatory analyses. No interim analysis of treatment effects will be used to determine whether recruitment continues or stops.
Sample size (or number of clusters) by treatment arms
Treatment arm: approximately 100 or more eligible leaders. Control arm: approximately 100 or more eligible leaders.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
No formal ex ante power calculation has been conducted. The planned sample size is based on feasibility within the project duration and the expected recruitment capacity among eligible participants with formal leadership responsibility. The study is intended to provide an informative test of the primary hypothesis, but it may be underpowered to detect small effects. Effect sizes and confidence intervals will therefore be reported and interpreted alongside statistical significance. Null findings will be interpreted as insufficient evidence of an effect within the limits of the achieved sample size, not as evidence that no effect exists.
Supporting Documents and Materials
IRB

Institutional Review Boards (IRBs)

IRB Name
University of Stavanger – HHUiS-IRB
IRB Approval Date
2026-06-15
IRB Approval Number
HHUiS-IRB-2026-005
Analysis Plan