|
Field
Trial Title
|
Before
Explaining AI-Generated Information: Implications for AI Adoption and Employee Preference for Supervisor Advice
|
After
The Impact of Explanatory Strategies on Employee Trust in Algorithmic Forecasts: An Experimental Investigation
|
|
Field
Abstract
|
Before
Algorithmically generated information is playing an increasingly central role in managers' decision-making processes. However, the value of these insights is only fully realized when they are effectively utilized. Consequently, the literature on algorithm aversion has explored the factors that influence their adoption. One underexplored phenomenon is managers' tendency to seek reassurance from their superiors. In our experiment, we investigate whether such advice-seeking behavior occurs and whether it can be influenced by providing explanations through explainable AI (XAI). We argue that the effectiveness of XAI in reducing excessive advice seeking depends on several factors: whether the algorithmically generated information aligns with the manager's expectations, and the type and quality of the explanation provided.
|
After
Algorithmically generated information is playing an increasingly central role in managers' decision-making processes. However, the value of these insights is only fully realized when they are effectively utilized. Consequently, the literature on algorithm aversion has explored the factors that influence their adoption. One underexplored phenomenon is managers' tendency to seek reassurance from their superiors. In our experiment, we investigate whether such advice-seeking behavior occurs and whether it can be influenced by providing explanations through explainable AI (XAI). We employ an online experiment involving crowd workers to isolate the effects of different XAI methods (Feature Importance Explanations vs. Counterfactual Explanations) on advice-seeking behavior.
|
|
Field
Trial Start Date
|
Before
March 04, 2025
|
After
March 11, 2025
|
|
Field
Trial End Date
|
Before
March 24, 2025
|
After
March 22, 2025
|
|
Field
JEL Code(s)
|
Before
M
|
After
C91, D91, M41
|
|
Field
Last Published
|
Before
March 04, 2025 05:49 AM
|
After
March 11, 2025 06:30 AM
|
|
Field
Intervention (Public)
|
Before
Using a publicly available dataset from Kaggle (https://www.kaggle.com/c/rossmann-store-sales/data), we develop an AI model to forecast store sales of a German convenience store chain. Additionally, we implement two different XAI methods producing i) Feature Importance Explanations and ii) Counterfactual Explanations along the AI forecasts for users. Forecasts are displayed in a format which is similar to contemporary business intelligence tools. Based on our experimental conditions, we deliberately select one store to present to our participants throughout the experiment. During the experiment, participants can interact with the dashboard to inform themselves about the store using the information provided by the system, the AI prediction and the explanation associated with their experimental condition. We will collect data from online crowd workers located in the US and Europe who are at least 18 years old. Each participant will be randomly assigned to one of our three experimental conditions.
|
After
Participants assume the role of a store manager in a convenience store chain. They are tasked with forecasting sales of the upcoming month using a sales development dashboard, which is similar to contemporary business intelligence tools. The assumed store chain uses AI to assist the store manager in the forecasting task. We implement two different explanatory strategies i) Feature Importance Explanations and ii) Counterfactual Explanations along the AI forecasts for users. During the experiment, participants can interact with the sales development dashboard to learn about the store using the information provided by the system, the AI prediction and the explanation associated with their experimental condition.
|
|
Field
Intervention Start Date
|
Before
March 04, 2025
|
After
March 11, 2025
|
|
Field
Intervention End Date
|
Before
March 24, 2025
|
After
March 22, 2025
|
|
Field
Primary Outcomes (Explanation)
|
Before
Advice seeking is a binary variable beeing one if participants seek advice from supervisors or zero otherwise.
|
After
|
|
Field
Experimental Design (Public)
|
Before
Participants will be setting sales targets (i.e., 10000€) for a fictious convenience store. They will be randomly divided into two experimental conditions, each subject to different conditions formulated by our independent variable:
• Algorithmic Advice (AI + Feature Importance Explanation vs. AI + Counterfactual Explanation)
On the welcome page of the experiment, we will inform participants about the procedure and potential exclusions through attention checks. Participant will be presented with a fictious target setting task. They are instructed to immerse themselves into the situation of setting targets for a convenience store. To further incentivize participants, they receive a bonus for setting accurate sales targets.
The experiment can be sectioned into the following steps:
1. Participants receive general information of their store, the stores past sales performance.
2. Participants indicate their sales expectations for the upcoming month.
3. Participants are provided an AI forecast (and a corresponding explanation) of expected sales for the upcoming month.
4. Based on the information provided beforehand, participants must set a sales target for their store.
5. The participants can choose to ask their supervisor for advice using an autonomous or dependent advice seeking logic.
6. Participants can readjust their previously selected sales target based on the information provided by their supervisor.
After completing the target setting process, participants will be asked to answer a questionnaire, to query their experience and general preferences. Finally, participants are asked to provide sociodemographic information.
|
After
Participants will be setting sales targets for a fictious convenience store. They will be randomly divided into three experimental conditions, each subject to different conditions formulated by our independent variable:
• Algorithmic Advice (AI vs. AI & Feature Importance Explanation vs. AI & Counterfactual Explanation)
On the welcome page of the experiment, we will inform participants about the procedure and potential exclusions through attention checks. They are instructed to immerse themselves into the situation of setting targets for a convenience store. To further incentivize participants, they receive a bonus for setting accurate sales targets.
The experiment can be sectioned into the following steps:
1. Participants receive general information of their store and the stores past sales performance.
2. Participants indicate their sales expectations for the upcoming month.
3. Participants are provided an AI forecast (and a corresponding explanation depending on the experimental condition) of expected sales for the upcoming month.
4. Based on the information provided beforehand, participants must set a sales target for their store.
5. The participants can choose to ask their supervisor for advice. However, advice is costly and reduces their potential bonus payment
6. Participants can readjust their previously selected sales target (based on the information provided by their supervisor).
After completing the target setting process, participants will be asked to answer a post experimental questionnaire, to query their experience and general preferences. Finally, participants are asked to provide sociodemographic information.
|
|
Field
Planned Number of Observations
|
Before
Our sample size will be calculated based on the results of our experimental pre-test using G-Power Analysis. Expected sample size: 200 participants
|
After
Our sample size will be calculated based on the results of a pre-test using Stata. Expected sample size: 540 participants.
|
|
Field
Additional Keyword(s)
|
Before
Advice-Seeking
Explainable Artificial Intelligence (XAI)
Target Setting
|
After
Explainable Artificial Intelligence (XAI)
Explanatory Strategies
Algorithmic Forecast
Advice Seeking
|
|
Field
Intervention (Hidden)
|
Before
The intervention is designed to investigate the following Hypotheses:
H1A: XAI methods reduce advice seeking.
H1B: This decrease is more pronounced when AI predictions deviate from user expectations.
H2: Counterfactual explanations are more effective in reducing advice seeking than feature importance explanations.
|
After
The intervention is designed to investigate the following Hypotheses:
Explanations reduce advice-seeking behavior.
Explanations increase advice-seeking behavior.
Counterfactual Explanations are more effective as compared to Feature Importance Explanations.
|
|
Field
Secondary Outcomes (End Points)
|
Before
Confidence in participants own prediction
Confidence in and weight of supervisor advice
Confidence in and weight of AI advice
|
After
Weight of AI advice
|
|
Field
Secondary Outcomes (Explanation)
|
Before
Confidence is measured using two 7-point-Likert-Scales adapted from Dietvorst et al. (2015).
Weight of supervisor advice and weight of AI advice is constructed using participants adjustments after receiving advice from supervisor and AI.
|
After
|