No, this is not how it’s supposed to be: The impact of XAI on ML model development

Last registered on March 26, 2025

Pre-Trial

Trial Information

General Information

Title
No, this is not how it’s supposed to be: The impact of XAI on ML model development
RCT ID
AEARCTR-0015506
Initial registration date
March 15, 2025

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
March 26, 2025, 8:15 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Region

Primary Investigator

Affiliation
Goethe Universität Frankfurt

Other Primary Investigator(s)

PI Affiliation
PI Affiliation
PI Affiliation

Additional Trial Information

Status
In development
Start date
2025-04-03
End date
2025-04-11
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
Explainable artificial intelligence (XAI) methods not only help to understand how machine learning models are working during operation, but also during the development of the models.
For instance, XAI can support machine learning developers in various tasks, such as debugging errors, identifying biases, and gaining an overall understanding of the data patterns identified by the ML algorithm.
As a result, providing explanations of model behavior may encourage developers to think more critically about the observed data patterns and model architecture.
However, this XAI-induced critical reflection can also have unintended consequences by triggering common human biases.
For example, we argue that when developers are presented with explanations of model behavior, they may compare these explanations with their mental models of the relationships among involved variables.
If discrepancies arise, this misalignment can create cognitive dissonance, potentially leading developers to reject high-performing models simply because they do not align with their expectations.
This study explores the following research questions:
Does the provision of explanations cause developers to impose their preconceived mental model on the ML model?
If so, does this alignment come at the expense of decreased model performance?
Finally, how does the use of XAI during development influence developers' confidence and their perception of the model’s value?
Our findings will contribute to a nuanced understanding of the role of XAI in the machine learning development lifecycle.
External Link(s)

Registration Citation

Citation
Bauer, Kevin et al. 2025. "No, this is not how it’s supposed to be: The impact of XAI on ML model development." AEA RCT Registry. March 26. https://doi.org/10.1257/rct.15506-1.0
Experimental Details

Interventions

Intervention(s)
Intervention (Hidden)
Intervention Start Date
2025-04-03
Intervention End Date
2025-04-11

Primary Outcomes

Primary Outcomes (end points)
1. Psychological ownership of participants toward developed machine learning models
2. Shift in participants' expectations of feature impacts (delta between pre-development and post-development mental models)
3. Performance expectations of the developed ML models
4. Actual prediction performance of the ML models
Primary Outcomes (explanation)
Psychological ownership of participants toward developed machine learning models:
Measures the extent to which participants feel a sense of personal investment, control, and responsibility over the machine learning models they develop.

Shift in participants' expectations of feature impacts (delta between pre-development and post-development mental models):
Assesses how participants' understanding of feature importance changes throughout the development process, comparing their initial and final mental models.

Performance expectations of the developed ML models:
Captures participants' subjective beliefs about how well their developed machine learning models will perform, including accuracy and reliability.

Actual prediction performance of the ML models:
Evaluates the objective effectiveness of the developed models by measuring their predictive accuracy and overall performance on relevant test data.

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
We aim to investigate these research questions in the context of an experiment that is embedded in
a realistic machine learning prediction task. Participating professional software developers have the
task to develop an ML model for real estate price estimations using a provided proprietary dataset.
The experimental design involves four groups differentiated by (i) the availability of pre-defined
XAI tools and (2) the availability of own mental models in the context of the prediction task. We
administer the XAI treatment by providing participants with a pre-defined, easy-to-use XAI function
that automatically shows different kinds of intuitive model explanations.
To control the availability of mental models about the prediction task, we mask the feature names so
that participants cannot infer the meaning of the original features. To prevent participants to infer
the meaning of the original variables from the variable distributions, we further normalize the values
of our dataset. We apply the normalization consistently across all experimental groups to maintain
comparability.

The experiment will be conducted by delegating prediction tasks to developers via Amazon MTurk, allowing participants to engage remotely. We developed the user interface of the task explanation, survey-based data collection, and distribution of datasets and further materials using the oTree framework, developed in Python (Chen et al., 2016), to which MTurk participants are forwarded. Upon entering the Otree environment, participants are randomly assigned to one of the four previously explained experimental groups using the itertools.cycle() function in Python, ensuring individual random assignment. Although the online setting reduces control over participants, it better simulates real-world data challenges and fosters more natural developer behavior. The task involves predicting apartment prices based on a proprietary dataset containing various features such as the number of rooms or construction year of the apartment. We compiled the dataset by scraping 5090 apartment listings from a large online platform, focusing on the seven largest cities in Germany during 2022. The dataset contains different features, such as the listing price per square meter, construction year, and the presence of balconies and basements. We further augmented the data with third-party information, namely the percentage of Green Party voters and unemployment rates. Before starting the prediction task, we ask participants to sort the available features in descending order of their perceived importance for the apartment price predictions. This task serves to elicit participants prior beliefs about the influence of each variable on apartment prices, which we refer to as their mental model. Participants will indicate whether a variable has a positive or negative impact and the approximate magnitude of this impact. Within the Otree environment, each participant will receive a Github repository containing basic instructions for the prediction task and a link to a Google Colab notebook. Participants will develop their models within this notebook and submit their final solutions. We have prepared four distinct GitHub repositories, each corresponding to a different experimental group to which participants were assigned earlier. In conditions involving XAI (Group A1 and B1, see table 1), we have incorporated a comprehensive XAI function directly into the Colab notebook. Participants can easily call this function by providing their trained prediction model along with the train and test data as inputs. This function automatically computes importance scores for all features in the form of SHAP values. SHAP values are a well-suited XAI method for our task as they provide reliable local and global feature importance-based explanations using intuitive visualizations. The function generates an interactive dashboard in the Colab code cell output frame. Within this dashboard, participants can activate three different SHAP visualizations: the bar plot, beeswarm plot, and interaction plot (Ponce-Bobadilla et al., 2024). Furthermore, we provide explanation narratives based on these SHAP values inspired by Martens et al. (2025). We introduce the function in the first Colab cell with a text summarizing all important information and guidelines for using it. Participants will have 10 days to complete the data challenge. After the experiment, participants will complete a post-experiment survey assessing their confidence in their submitted models and other secondary measures. This subjective measure will provide insights into how different treatment conditions affect participants’ perceived performance. To assess possible changes of the mental models, we ask participants again to sort the available features in descending order of their perceived importance. We will begin with a pilot study of 10 participants to refine the experimental design and materials. Afterward, the study will scale to approximately 200 participants, with 50 per treatment group, ensuring sufficient statistical power. Participants will be compensated above U.S. minimum wage, and the top performer in each group, evaluated on a hold-out dataset, will receive an additional prize payment.

Experimental Design Details
Randomization Method
Upon entering the Otree environment, participants are randomly assigned to one of the four previously explained experimental groups using the itertools.cycle() function in Python, ensuring individual random assignment.
Randomization Unit
Individual randomization
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
Up to 200 professional machine learning developers
Sample size: planned number of observations
Up to 200 professional machine learning developers
Sample size (or number of clusters) by treatment arms
50 participants for each of the four experimental units
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Supporting Documents and Materials

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
IRB

Institutional Review Boards (IRBs)

IRB Name
Gemeinsame Ethikkommission Wirtschaftswissenschaften
IRB Approval Date
2025-03-06
IRB Approval Number
N/A

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials