A Design Centric View of AI Enhanced Decision Making

Interventions

In this study, participants will be randomly assigned to one of 12 experimental groups (blocked minimized randomization, due to the different levels of difficulty and therefore attrition rate across arms) to examine the effects of different types of AI assistance on decision-making and the development of theories.



Experimental Arms (12 arms)

Plausible Challenge (An easier to solve problem)

1. Control – Single Hypothesis (Group 1a): No Additional Assistance

Intervention: Participants will receive a plausible decision problem and have a single text box to enter their theorizing without any additional assistance.

Objective: Baseline measure of theorizing with a constrained single hypothesis.

2. Control – Multiple Hypotheses (Group 1b): No Additional Assistance

Intervention: Participants will receive a plausible decision problem and can add multiple text boxes to develop several theories without additional assistance.

Objective: Measure the impact of generating multiple hypotheses on theorizing without external assistance.

3. General AI Assistance – Single Hypothesis (Group 2a): Access to ChatGPT

Intervention: Participants will use ChatGPT O3-mini for assistance in brainstorming and refining their theorizing in response to a plausible problem, limited to one hypothesis.

Objective: Evaluate general AI impact on theorizing when constrained to a single hypothesis.

4. General AI Assistance – Multiple Hypotheses (Group 2b): Access to ChatGPT

Intervention: Participants will use ChatGPT O3-mini to develop and refine multiple hypotheses, having multiple text boxes available for entry.

Objective: Assess how general AI assistance combined with multiple hypotheses affects theory.

5. Specialized Agentic AI Assistance – Single Hypothesis (Group 3a): Access to agentic AI (TBV-trained AI)

Intervention: Participants will have access to agentic AI to generate and refine a single theory aligned with the theory-based view.

Objective: Investigate the specialized AI's impact on theorizing when limited to one hypothesis.

6. Specialized Agentic AI Assistance – Multiple Hypotheses (Group 3b): Access to agentic AI (TBV-trained AI)

Intervention: Participants will use agentic AI to generate, refine, and potentially autogenerate multiple theories.

Objective: Evaluate the effectiveness of specialized AI support in enhancing theorizing through multiple hypothesis generation.

Implausible Challenge (A more difficult to address problem)

7. Control – Single Hypothesis (Group 4a): No Additional Assistance

Intervention: Participants will receive an implausible decision problem and enter their theory using a single text box without assistance.

Objective: Baseline measure for implausible problems with a single hypothesis constraint.

8. Control – Multiple Hypotheses (Group 4b): No Additional Assistance

Intervention: Participants will tackle an implausible problem, generating multiple hypotheses in multiple text boxes without external assistance.

Objective: Examine how multiple hypotheses generation without assistance influences theorizing in challenging scenarios.

9. General AI Assistance – Single Hypothesis (Group 5a): Access to ChatGPT

Intervention: Participants facing an implausible problem will use ChatGPT O3-mini to assist in refining a single hypothesis.

Objective: Assess the impact of general AI assistance in challenging scenarios when constrained to one hypothesis.

10. General AI Assistance – Multiple Hypotheses (Group 5b): Access to ChatGPT

Intervention: Participants will leverage ChatGPT O3-mini to brainstorm, refine, and develop multiple theories for an implausible problem.

Objective: Evaluate the advantage of general AI support combined with multiple hypothesis generation in difficult problem settings.

11. Specialized Agentic AI Assistance – Single Hypothesis (Group 6a): Access to agentic AI (TBV-trained AI)

Intervention: Participants will use agentic AI to generate and refine one theory in response to an implausible problem.

Objective: Determine specialized AI's effectiveness in enhancing theorizing under single-hypothesis constraints in difficult scenarios.

12. Specialized Agentic AI Assistance – Multiple Hypotheses (Group 6b): Access to agentic AI (TBV-trained AI)

Intervention: Participants will utilize agentic AI to generate and refine multiple hypotheses, including autogenerated theories, in response to an implausible problem.

Objective: Investigate how specialized AI assistance maximizes theory development effectiveness through multiple hypothesis generation in complex contexts.



Purpose of Interventions

The interventions are designed to assess the impact of different types of AI assistance on prior formation, specifically focusing on:

The number of hypotheses n

The quality of the theory z

The confidence in the theory ω

Expected Success Probability π

And interacting these effects with the plausibility or implausibility of the problem.

By comparing these outcomes across the three groups, the study aims to provide insights into how various forms of AI assistance can influence theories and decision-making. This understanding is valuable for organizations considering the integration of AI support tools to enhance decision-making and innovation

Interventions

In this study, participants will be randomly assigned to one of 12 experimental groups (blocked minimized randomization, due to the different levels of difficulty and therefore attrition rate across arms) to examine the effects of different types of AI assistance on decision-making and the development of theories.



Experimental Arms (12 arms)

Plausible Challenge (An easier to solve problem)

1. Control – Single Hypothesis (Group 1a): No Additional Assistance

Intervention: Participants will receive a plausible decision problem and have a single text box to enter their theorizing without any additional assistance.

Objective: Baseline measure of theorizing with a constrained single hypothesis.

2. Control – Multiple Hypotheses (Group 1b): No Additional Assistance

Intervention: Participants will receive a plausible decision problem and can add multiple text boxes to develop several theories without additional assistance.

Objective: Measure the impact of generating multiple hypotheses on theorizing without external assistance.

3. General AI Assistance – Single Hypothesis (Group 2a): Access to ChatGPT

Intervention: Participants will use ChatGPT O3-mini for assistance in brainstorming and refining their theorizing in response to a plausible problem, limited to one hypothesis.

Objective: Evaluate general AI impact on theorizing when constrained to a single hypothesis.

4. General AI Assistance – Multiple Hypotheses (Group 2b): Access to ChatGPT

Intervention: Participants will use ChatGPT O3-mini to develop and refine multiple hypotheses, having multiple text boxes available for entry.

Objective: Assess how general AI assistance combined with multiple hypotheses affects theory.

5. Specialized Agentic AI Assistance – Single Hypothesis (Group 3a): Access to agentic AI (TBV-trained AI)

Intervention: Participants will have access to agentic AI to generate and refine a single theory aligned with the theory-based view.

Objective: Investigate the specialized AI's impact on theorizing when limited to one hypothesis.

6. Specialized Agentic AI Assistance – Multiple Hypotheses (Group 3b): Access to agentic AI (TBV-trained AI)

Intervention: Participants will use agentic AI to generate, refine, and potentially autogenerate multiple theories.

Objective: Evaluate the effectiveness of specialized AI support in enhancing theorizing through multiple hypothesis generation.

Implausible Challenge (A more difficult to address problem)

7. Control – Single Hypothesis (Group 4a): No Additional Assistance

Intervention: Participants will receive an implausible decision problem and enter their theory using a single text box without assistance.

Objective: Baseline measure for implausible problems with a single hypothesis constraint.

8. Control – Multiple Hypotheses (Group 4b): No Additional Assistance

Intervention: Participants will tackle an implausible problem, generating multiple hypotheses in multiple text boxes without external assistance.

Objective: Examine how multiple hypotheses generation without assistance influences theorizing in challenging scenarios.

9. General AI Assistance – Single Hypothesis (Group 5a): Access to ChatGPT

Intervention: Participants facing an implausible problem will use ChatGPT O3-mini to assist in refining a single hypothesis.

Objective: Assess the impact of general AI assistance in challenging scenarios when constrained to one hypothesis.

10. General AI Assistance – Multiple Hypotheses (Group 5b): Access to ChatGPT

Intervention: Participants will leverage ChatGPT O3-mini to brainstorm, refine, and develop multiple theories for an implausible problem.

Objective: Evaluate the advantage of general AI support combined with multiple hypothesis generation in difficult problem settings.

11. Specialized Agentic AI Assistance – Single Hypothesis (Group 6a): Access to agentic AI (TBV-trained AI)

Intervention: Participants will use agentic AI to generate and refine one theory in response to an implausible problem.

Objective: Determine specialized AI's effectiveness in enhancing theorizing under single-hypothesis constraints in difficult scenarios.

12. Specialized Agentic AI Assistance – Multiple Hypotheses (Group 6b): Access to agentic AI (TBV-trained AI)

Intervention: Participants will utilize agentic AI to generate and refine multiple hypotheses, including autogenerated theories, in response to an implausible problem.

Objective: Investigate how specialized AI assistance maximizes theory development effectiveness through multiple hypothesis generation in complex contexts.



Purpose of Interventions

The interventions are designed to assess the impact of different types of AI assistance on prior formation, specifically focusing on:

The number of hypotheses n

The quality of the theory z

The confidence in the theory ω

Expected Success Probability π

And interacting these effects with the plausibility or implausibility of the problem.

By comparing these outcomes across the three groups, the study aims to provide insights into how various forms of AI assistance can influence theories and decision-making. This understanding is valuable for organizations considering the integration of AI support tools to enhance decision-making and innovation

2025-04-21

2025-04-22

Main Outcomes:
Expected Success Probability (π)


Mechanism Variables:
Number of Hypotheses (n)
Theory Quality (z)
Confidence in Theory (ω)
Confidence in π

Main Outcomes:
Expected Success Probability (π): Participants' estimated likelihood of successfully achieving the end goal of the decision problem based on their developed theories.



Mechanism Variables:
Number of Hypotheses (n): The total count of distinct hypotheses generated by participants.
Theory Quality (z): Evaluations of clarity, coherence, and strategic alignment of the theories developed (Based on reasoning elements, reasoning logic)
Confidence in Theory (ω): Self-reported confidence in the validity and applicability of theories.
Confidence in π: Self-reported confidence in the reported subjective probability of success

Might include other proxies for the potential plausibility and value of the solutions to the decision challenge, such as:
* Novelty
* Strategic Viability
* Financial Value
* Environmental Value
* Quality of each solution based on the previous 4

These variables come from Boussioux et al. 2024

This study utilizes a randomized controlled experimental design to examine the effects of different types of AI assistance on managerial decision-making and theory development. Participants will be randomly assigned to one of six experimental conditions, varying by the type of AI assistance (none, general AI, specialized AI) and the plausibility of the decision-making problem (plausible vs. implausible).



Participants

Recruitment:

Participants will be recruited via Prolific, focusing on individuals with managerial experience. Efforts will be made to achieve balanced gender representation and diversity across educational backgrounds and industries.



Participants will be randomly assigned to one of six experimental groups:



Plausible Challenge

Control – Single Hypothesis (Group 1a): No Additional Assistance

Control – Multiple Hypotheses (Group 1b): No Additional Assistance

General AI Assistance – Single Hypothesis (Group 2a): Access to ChatGPT

General AI Assistance – Multiple Hypotheses (Group 2b): Access to ChatGPT

Specialized Agentic AI Assistance – Single Hypothesis (Group 3a): Access to agentic AI (TBV-trained AI)

Specialized Agentic AI Assistance – Multiple Hypotheses (Group 3b): Access to agentic AI (TBV-trained AI)

Implausible Challenge

Control – Single Hypothesis (Group 4a): No Additional Assistance

Control – Multiple Hypotheses (Group 4b): No Additional Assistance

General AI Assistance – Single Hypothesis (Group 5a): Access to ChatGPT

General AI Assistance – Multiple Hypotheses (Group 5b): Access to ChatGPT

Specialized Agentic AI Assistance – Single Hypothesis (Group 6a): Access to agentic AI (TBV-trained AI)

Specialized Agentic AI Assistance – Multiple Hypotheses (Group 6b): Access to agentic AI (TBV-trained AI)



Procedure

All participants will first consent to participate and complete a baseline survey gathering demographic data (age, gender, education level, education field, industry, job function, experience, managerial experience).



Introduction Video:

Participants will watch an introductory video covering

The Theory-Based View (TBV)

The role of theories in strategic decision-making.

How theories contribute to value creation and capture.



Decision Problem Presentation:

Participants will be presented with either a plausible or implausible decision-making problem, based on their assigned group, designed to stimulate strategic thinking.



Group-Specific Interventions:

Plausible Challenge

Groups 1a & 1b (Control - Human Only):

1a (Single hypothesis): Participants generate one theory without any AI assistance.

1b (Multiple hypotheses): Participants generate multiple theories without any AI assistance.

Objective: Establish a baseline for theory quality, novelty, and confidence.

Groups 2a & 2b (General AI - ChatGPT):

2a (Single hypothesis): Participants generate one theory with ChatGPT assistance.

2b (Multiple hypotheses): Participants generate multiple theories with ChatGPT assistance.

Objective: Assess the effect of general AI assistance on theory development quality, novelty, and confidence.

Groups 3a & 3b (Specialized Agentic AI - Aristotle):

3a (Single hypothesis): Participants generate one theory with specialized Aristotle AI assistance.

3b (Multiple hypotheses): Participants generate multiple theories, with the option to autogenerate theories using Aristotle.

Objective: Evaluate the enhanced impact of specialized AI on theory quality, novelty, and confidence.

Implausible Challenge

Groups 4a & 4b (Control - Human Only):

4a (Single hypothesis): Participants generate one theory without any AI assistance.

4b (Multiple hypotheses): Participants generate multiple theories without any AI assistance.

Objective: Baseline measurement for challenging decision scenarios.

Groups 5a & 5b (General AI - ChatGPT):

5a (Single hypothesis): Participants generate one theory with ChatGPT assistance.

5b (Multiple hypotheses): Participants generate multiple theories with ChatGPT assistance.

Objective: Investigate general AI’s effect on theory development quality, novelty, and confidence in challenging scenarios.

Groups 6a & 6b (Specialized Agentic AI - Aristotle):

6a (Single hypothesis): Participants generate one theory with Aristotle AI assistance.

6b (Multiple hypotheses): Participants generate multiple theories, including autogenerated options from Aristotle AI.

Objective: Determine the specialized AI's capacity to enhance theory quality, novelty, and confidence in complex decision-making contexts.





Theory Development Task:

Participants will:

Reflect on the given decision problem.

Generate and develop their theories using the provided assistance.

Submit detailed explanations of their approaches and solutions.



Post-Treatment Questionnaire 1:

Participants will again assess:

Confidence: Level of confidence demonstrated in developed theories.

Expected Probability of Success (π): Evaluated based on theoretical explanations and justifications provided by participants.

Self-evaluation of the 5 quality measures developed by Lakhani et al. 2024



Post-Treatment Questionnaire 2:

Participants will answer questions regarding:

Some of the question in the baseline survey, including AI aversion, and GPT familiarity

Knowledge Breadth

Knowledge Depth

Awareness of the Unknown (Unforeseen contingencies)

Participants will receive a debrief that includes:

An explanation of the study's purpose.

The nature of the experimental conditions.

Information about how their data will be used.

Contact information for any follow-up questions.



Evaluation

A specially instructed AI agents, blind to treatment conditions, will evaluate submissions on:

Quality:

Logical coherence (on the reasoning elements and logical reasoning)

Strategic relevance and practical applicability, using the 5 quality measures developed by Lakhani et al. 2024

Expected Probability of Success (π) and the confidence on it:

Evaluated based on theoretical explanations and justifications provided by participants.



And we count the number of theories the user enters in the experiment.

This study utilizes a randomized controlled experimental design to examine the effects of different types of AI assistance on managerial decision-making and theory development. Participants will be randomly assigned to one of six experimental conditions, varying by the type of AI assistance (none, general AI, specialized AI) and the plausibility of the decision-making problem (plausible vs. implausible).



Participants

Recruitment:

Participants will be recruited via Prolific, focusing on individuals with managerial experience. Efforts will be made to achieve balanced gender representation and diversity across educational backgrounds and industries.



Participants will be randomly assigned to one of six experimental groups:



Plausible Challenge

Control – Single Hypothesis (Group 1a): No Additional Assistance

Control – Multiple Hypotheses (Group 1b): No Additional Assistance

General AI Assistance – Single Hypothesis (Group 2a): Access to ChatGPT

General AI Assistance – Multiple Hypotheses (Group 2b): Access to ChatGPT

Specialized Agentic AI Assistance – Single Hypothesis (Group 3a): Access to agentic AI (TBV-trained AI)

Specialized Agentic AI Assistance – Multiple Hypotheses (Group 3b): Access to agentic AI (TBV-trained AI)

Implausible Challenge

Control – Single Hypothesis (Group 4a): No Additional Assistance

Control – Multiple Hypotheses (Group 4b): No Additional Assistance

General AI Assistance – Single Hypothesis (Group 5a): Access to ChatGPT

General AI Assistance – Multiple Hypotheses (Group 5b): Access to ChatGPT

Specialized Agentic AI Assistance – Single Hypothesis (Group 6a): Access to agentic AI (TBV-trained AI)

Specialized Agentic AI Assistance – Multiple Hypotheses (Group 6b): Access to agentic AI (TBV-trained AI)



Procedure

All participants will first consent to participate and complete a baseline survey gathering demographic data (age, gender, education level, education field, industry, job function, experience, managerial experience).



Introduction Video:

Participants will watch an introductory video covering

The Theory-Based View (TBV)

The role of theories in strategic decision-making.

How theories contribute to value creation and capture.



Decision Problem Presentation:

Participants will be presented with either a plausible or implausible decision-making problem, based on their assigned group, designed to stimulate strategic thinking.



Group-Specific Interventions:

Plausible Challenge

Groups 1a & 1b (Control - Human Only):

1a (Single hypothesis): Participants generate one theory without any AI assistance.

1b (Multiple hypotheses): Participants generate multiple theories without any AI assistance.

Objective: Establish a baseline for theory quality, novelty, and confidence.

Groups 2a & 2b (General AI - ChatGPT):

2a (Single hypothesis): Participants generate one theory with ChatGPT assistance.

2b (Multiple hypotheses): Participants generate multiple theories with ChatGPT assistance.

Objective: Assess the effect of general AI assistance on theory development quality, novelty, and confidence.

Groups 3a & 3b (Specialized Agentic AI - Aristotle):

3a (Single hypothesis): Participants generate one theory with specialized Aristotle AI assistance.

3b (Multiple hypotheses): Participants generate multiple theories, with the option to autogenerate theories using Aristotle.

Objective: Evaluate the enhanced impact of specialized AI on theory quality, novelty, and confidence.

Implausible Challenge

Groups 4a & 4b (Control - Human Only):

4a (Single hypothesis): Participants generate one theory without any AI assistance.

4b (Multiple hypotheses): Participants generate multiple theories without any AI assistance.

Objective: Baseline measurement for challenging decision scenarios.

Groups 5a & 5b (General AI - ChatGPT):

5a (Single hypothesis): Participants generate one theory with ChatGPT assistance.

5b (Multiple hypotheses): Participants generate multiple theories with ChatGPT assistance.

Objective: Investigate general AI’s effect on theory development quality, novelty, and confidence in challenging scenarios.

Groups 6a & 6b (Specialized Agentic AI - Aristotle):

6a (Single hypothesis): Participants generate one theory with Aristotle AI assistance.

6b (Multiple hypotheses): Participants generate multiple theories, including autogenerated options from Aristotle AI.

Objective: Determine the specialized AI's capacity to enhance theory quality, novelty, and confidence in complex decision-making contexts.





Theory Development Task:

Participants will:

Reflect on the given decision problem.

Generate and develop their theories using the provided assistance.

Submit detailed explanations of their approaches and solutions.



Post-Treatment Questionnaire 1:

Participants will again assess:

Confidence: Level of confidence demonstrated in developed theories.

Expected Probability of Success (π): Evaluated based on theoretical explanations and justifications provided by participants.

Self-evaluation of the 5 quality measures developed by Lakhani et al. 2024



Post-Treatment Questionnaire 2:

Participants will answer questions regarding:

Some of the question in the baseline survey, including AI aversion, and GPT familiarity

Knowledge Breadth

Knowledge Depth

Awareness of the Unknown (Unforeseen contingencies)

Participants will receive a debrief that includes:

An explanation of the study's purpose.

The nature of the experimental conditions.

Information about how their data will be used.

Contact information for any follow-up questions.



Evaluation

A specially instructed AI agents, blind to treatment conditions, will evaluate submissions on:

Quality:

Logical coherence (on the reasoning elements and logical reasoning)

Strategic relevance and practical applicability, using the 5 quality measures developed by Lakhani et al. 2024

Expected Probability of Success (π) and the confidence on it:

Evaluated based on theoretical explanations and justifications provided by participants.



And we count the number of theories the user enters in the experiment.

To allocate participants to experimental conditions, we use minimized blocked randomization (batch size = 50) based on three covariates: a) education level and field; b) AI aversion; and c) level of self-confidence. This provides the baseline allocation of participants to the experimental conditions.

To ensure a balanced number of participants in each group and to manage the flow of the experiment, we limit the number of participants who can enter the experimental platform at the same time. This helps prevent an imbalance at the end of the experiment caused by participants who start but do not finish the experiment.

Individuals

No

No clusters

2500 managers from Prolific

208

The target sample size is about 2500 participants, and 208 in each arm. We justify the size of the samples based on experimental power calculations conducted using G*Power 3.1.9.7, command t-tests, Means: Difference between two independent means (two groups). We assume we detect a medium effect size d=0.25 calculated using a pilot study with 240 participants. We assume standard type I and type II errors (α err prob = 0.05 Power (1-β err prob) = 0.80), with an allocation ratio of 1. This results in 199 observations per treatment arm, which results in a minimum of 2388 observation in total. As such, we round this rough approximation up to 2500 observations, to ensure statistical relevance.