You need to sign in or sign up before continuing.

Preference for Explainable AI Experiment

Last registered on March 19, 2025

View Trial History

Pre-Trial

Trial Information

General Information

Title

Preference for Explainable AI Experiment

RCT ID

AEARCTR-0015581

Initial registration date

March 16, 2025

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published

March 19, 2025, 11:56 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Country

United States of America

Region

Primary Investigator

Name

Alex Chan

Affiliation

Harvard University

Contact Primary Investigator

Other Primary Investigator(s)

Additional Trial Information

Status

In development

Start date

2025-03-18

End date

2025-03-22

Keywords

Behavior, Lab

Additional Keywords

XAI, Interpretability, algorithms, artificial intelligence, moral wiggle room, algorithmic fairness

JEL code(s)

B4, C1, C91, C92, D81

Secondary IDs

Prior work

This trial does not extend or rely on any prior RCTs.

Abstract

Lab experiment to investigate individual preference for getting information generated by AI and the explanation of how the AI made that prediction. The experiment is done in a real-stakes decision where a Black-Box AI informs a decision to allocate an actual US$10, 000-loan.

External Link(s)

Registration Citation

Citation

Chan, Alex. 2025. "Preference for Explainable AI Experiment." AEA RCT Registry. March 19. https://doi.org/10.1257/rct.15581-1.0

Sponsors & Partners

Experimental Details

Interventions

Intervention(s)

Individuals make a decision to determine how a private lender allocate a $10000-loan. Participants are randomized into a neutral treatment where payoffs are not tied to actual loan repayment outcome, or a "lender-aligned" treatment where they have with direct stakes tied to loan repayment.

Intervention (Hidden)

2X3X5X3 design
2: neutral treatment where payoffs are not tied to actual loan repayment outcome, or a "lender-aligned" treatment where they have with direct stakes tied to loan repayment
3: race-gender of prospective borrowers revealed to be different AND no decision to choose whether to see predicted default risk or not, race-gender of prospective borrowers NOT revealed AND no decision to choose whether to see predicted default risk or not, race-gender of prospective borrowers revealed to be different AND with decision to choose whether to see predicted default risk or not
5: always see explanation of AI, never see explanation of AI, option to see explanation of AI with no salience, option to see explanation of AI with hint that financial factors might be used by AI, option to see explanation of AI with hint that financial factors and demographics might be used by AI
3: 3 versions of explanation for those who see or chose to see explanation. One vague description of how neural network AI risk prediction works, one that plus a SHAP interpretation of why the high risk borrower was deemed as such just based on financials, one that plus a SHAP interpretation of why the high risk borrower was deemed as such just based on financials AND race and gender

Intervention Start Date

2025-03-18

Intervention End Date

2025-03-22

Primary Outcomes

Primary Outcomes (end points)

(1) Binary decision of whether the participant want to see an explanation of how the AI made the prediction of the default risk of the borrowers before making a loan allocation decision
(2) Binary decision of whether the participant want to see the predicted default risk of the borrowers before making a loan allocation decision

Primary Outcomes (explanation)

Both study information-avoidance decisions.

Secondary Outcomes

Secondary Outcomes (end points)

(3) Whether the participant want to allocate 90% of the loan to the low-risk borrower and 10% to the high-risk borrower, or 50% to each

Secondary Outcomes (explanation)

This gets at whether the information changes the actual loan allocation decision (i.e. to see if the information influences decision)

Experimental Design

I experimentally test how individual's preference for information regarding an AI is influenced by certain factors.

Experimental Design Details

3 screens for bots and one attention check and one verifying comprehension test are included.
The final analysis sample and results to be reported in the paper will only include the subjects that passed ALL 3 bot screens (CAPTCHA, honey-pot question visible to bot but not to humans, CAPTCHA + timing), and passed the attention check (selected both "Strongly Agree" AND "Strongly Disagree"), as well as those who answered that "Determine how the private lender will allocate the $10,000 loan between the two real borrowers" in the comprehension test.
The full sample will be analyzed to show robustness, but the main analysis will be based on the sample of people who passed all tests listed above here.

Randomization Method

Randomization done using Qualtrics' in-built function

Randomization Unit

individual

Was the treatment clustered?

Experiment Characteristics

Sample size: planned number of clusters

See below

Sample size: planned number of observations

2,500

Sample size (or number of clusters) by treatment arms

500 for non-lender-aligned, 500 for lender-aligned for main arm. Same numbers for a variant of the main arm where race-gender descriptors for borrowers are replaced by odd/even number birth-day/month.
250 for non-lender-aligned, 250 for lender-aligned for an arm similar to main arm except that participants also have a choice to see the predicted default risk generated by AI or not.
2500 in total.

Minimum detectable effect size for main outcomes (accounting for sample design and clustering)

Supporting Documents and Materials

IRB