LLMs in Strategic Decision Making: Problem Formulation and Solution Ideation

Last registered on February 16, 2024

View Trial History

Pre-Trial

Trial Information

General Information

Title

LLMs in Strategic Decision Making: Problem Formulation and Solution Ideation

RCT ID

AEARCTR-0013044

Initial registration date

February 15, 2024

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published

February 16, 2024, 4:18 PM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Primary Investigator

Name

Hyunjin Kim

Affiliation

Harvard University

Contact Primary Investigator

Other Primary Investigator(s)

PI Name

Nety Wu

PI Affiliation

INSEAD

Contact Investigator

PI Name

Chengyi Lin

PI Affiliation

INSEAD

Contact Investigator

Additional Trial Information

Status

In development

Start date

2024-02-15

End date

2025-02-15

Keywords

Behavior, Firms & Productivity

Additional Keywords

Strategic decision-making

JEL code(s)

Secondary IDs

Prior work

This trial does not extend or rely on any prior RCTs.

Abstract

This project aims to understand the influence of employing a Large Language Model (LLM) on problem formulation and solution ideation stages during the strategic decision making process.

External Link(s)

Registration Citation

Citation

Kim, Hyunjin, Chengyi Lin and Nety Wu. 2024. "LLMs in Strategic Decision Making: Problem Formulation and Solution Ideation." AEA RCT Registry. February 16. https://doi.org/10.1257/rct.13044-1.0

Sponsors & Partners

Experimental Details

Interventions

Intervention(s)

Intervention Start Date

2024-02-15

Intervention End Date

2024-02-26

Primary Outcomes

Primary Outcomes (end points)

All participants will be asked to articulate a problem statement and their thought process in developing the problem statement. We will analyze the length, depth, and breadth of this problem statement, and how they prioritize the problems, if they identify multiple. We will also categorize their response on the thought process using NLP techniques.
From the options they input, we will observe the following outcomes:
(1) the number of total options that the student brainstormed
(2) the number/share of strategic (rather than operational) options that the student brainstormed
(3) a binary measure of whether the options were mutually exclusive
(4) the number/share of options that suggest that the company continue its current strategy (vs. exit or expand)
(5) binary measures of whether the best-chosen option is to "continue", "exit", or "expand"
In addition, we will measure the time they used to complete the task, and the cognitive load of the participant using a math question at the end of the experiment.

Primary Outcomes (explanation)

Some of the measures will be coded by two independent coders using a rubric who are blind to the random assignment, e.g.:
(1) whether the options are strategic or operational
(2) whether the options are mutually exclusive
(3) whether the options suggest that the company continue its current strategy vs. exit or expand

Secondary Outcomes

Secondary Outcomes (end points)

Survey questions to all participants: (1) Describe how you developed your options; (2) How confident are you in the options you have developed? (3) How would you describe the difficulty of the task? (4) Frequency of using LLMs and familiarity with LLMs.
Additional survey questions to the treatment group only: (1) Did you find the LLM tool provided useful? (2) Please elaborate on why it was useful or not useful.
Peer evaluations of option quality, binary variables on how detailed the option is, various details about the nature of the option, and group-level outcomes on strategic options.
Secondary outcomes will be collected and coded based on feasibility (e.g., if budget allows).

Secondary Outcomes (explanation)

Experimental Design

This project aims to understand the influence of employing a Large Language Model (LLM) on problem formulation and solution ideation stages during the strategic decision making process.

Experimental Design Details

Not available

Randomization Method

Individuals are randomized into control and treatment by stratifying on gender and industry (whether they worked in consulting), using a computer.

Randomization Unit

Randomization is conducted at the individual level. Each participant is first randomly assigned to one of the three conditions, stratified on gender and industry (whether they worked in consulting).

Was the treatment clustered?