Causal Reasoning and AI-Assisted Decision Making

This trial investigates the independent and interactive effects of causal reasoning training and large language model (LLM) assistance on strategic decision-making. Using a 2×2 factorial design with first year bachelor students (N = 800), we examine how these cognitive enhancement approaches influence the analysis of complex business cases by participants. The experiment tests three primary hypotheses: (1) complementarity (interaction effects) of combined interventions, (2) main effect of causal reasoning training, and (3) main effect of ChatGPT access. Performance is assessed through rubric-based evaluation and comparison with expert solutions, with a two stage least squares analysis that examines the underlying mechanisms.

External Link(s)

Registration Citation

Citation

Asirvatham, Hemanth et al. 2025. "Causal Reasoning and AI-Assisted Decision Making." AEA RCT Registry. October 31. https://doi.org/10.1257/rct.17124-1.0

Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Experimental Details

Interventions

Intervention(s)

Intervention (Hidden)

The first dimension contrasts causal reasoning training with a placebo control: participants play either a causal reasoning game (featuring lessons, 12 causal-specific questions, and feedback) or a placebo game (same format and content but with generic questions, no lessons, no feedback). The second dimension provides access to ChatGPT through the standard web interface at chatgpt.com, allowing participants to interact with the AI assistant while solving the case.

Intervention Start Date

2025-10-29

Intervention End Date

2025-10-31

Primary Outcomes

Primary Outcomes (end points)

Primary preformance Measure and Secondary Performance Measure.

Primary Outcomes (explanation)

Primary performance measure: Derives from a structured performance rubric evaluating Alumni Merchandising Performance based on two dimensions: Awareness (strategies to ensure alumni notice and recall merchandise) and Usage (strategies driving purchasing behavior and en
gagement). Each dimension is scored 1-5, with the overall score calculated as (Awareness + Usage)/ 2.

Secondary performance measure: Calculated as the semantic distance between participant responses and a solution developed by experienced business strategists with specific expertise in marketing strategy. This complementary measure provides an alternative performance assessment less dependent on specific rubric criteria.

Secondary Outcomes

Secondary Outcomes (end points)

Mechanism, Process, and Behavioural Measures.

Secondary Outcomes (explanation)

Mechanism Measures
• Causal Score (3 attributes, each on a 1–5 scale), Coherent Logic, Falsifiability, Mechanism Identification
• Self-reported reasoning style
• Information augmentation (breadth/depth)
Process Measures
• Time allocation
• Solution iterations
• ChatGPT usage metrics
Behavioral Measures
• AI aversion
• AI automation bias

Experimental Design

The study implements a fully crossed factorial design with two treatment dimensions. The first dimension contrasts causal reasoning training with a placebo control. The second dimension provides access/non access to ChatGPT through the standard web interface at chatgpt.com. This design yields four experimental conditions: Placebo without ChatGPT, Placebo with ChatGPT, Treatment without ChatGPT, and Treatment with ChatGPT.

Experimental Design Details

The study implements a fully crossed factorial design with two treatment dimensions. The first dimension contrasts causal reasoning training with a placebo control: participants play either a causal reasoning game (featuring lessons, 12 causal-specific questions, and feedback) or a placebo game (same format and content but with generic questions, no lessons, no feedback). The second dimension provides access to ChatGPT through the standard web interface at chatgpt.com, allowing participants to interact with the AI assistant while solving the case. This design yields four experimental conditions: placebo game without ChatGPT, placebo game with ChatGPT, causal game without ChatGPT, and causal game with ChatGPT.

Randomization Method

We use simple randomization using the random assignment of students to classes in the enrollment at the start of the academic year: Randomizing classes into treatments preserves the integrity of the analysis (as it implies random assignment to experimental conditions) while respecting organizational constraints. The participants’ samples in the 4 experimental conditions are similar–on average–along a set of meaningful covariates, as confirmed by the balance checks that ensure that participants in the 4 experimental conditions do not significantly differ. Randomization done in office using Python.

Randomization Unit

First-year Bachelor's classes within three degree courses (CLEAM, BIEF, BIEM).

Was the treatment clustered?

Yes

Experiment Characteristics

Sample size: planned number of clusters

A total of 13 classes participate, representing all first-year students in the courses scheduled during the experimental period.

Sample size: planned number of observations

800 first-year bachelor students.

Sample size (or number of clusters) by treatment arms

3 classes for treatment condition Causal Training + No ChatGPT.
4 classes for treatment condition Causal Training + ChatGPT.
3 classes for treatment condition No Causal Training + No ChatGPT.
3 classes for treatment condition No Causal Training + ChatGPT.

Minimum detectable effect size for main outcomes (accounting for sample design and clustering)

With N = 800 (200 participants per cell), we have more than adequate power for all planned analyses. For the main effects assuming Cohen’s d = 0.3 (typical for educational interventions), the power exceeds 97%, using two-way ANOVA with α = 0.05. Pairwise comparisons between conditions maintain over 90% power for detecting small-to-medium effects (Cohen’s d = 0.3). Minimum detectable effect sizes as small as d = 0.20 still ensure 80% power. These calculations confirm that we can reliably detect educationally meaningful effects without wasteful over-recruitment.

Supporting Documents and Materials

IRB