The Effect of AI on Idea Generation and Evaluation

Last registered on May 21, 2025
View Trial History

Pre-Trial

Trial Information

General Information

Title
The Effect of AI on Idea Generation and Evaluation
RCT ID
AEARCTR-0016016
Initial registration date
May 14, 2025

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
May 21, 2025, 2:25 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Primary Investigator

Name
Mana Heshmati
Affiliation
University of Washington, Seattle
Contact Primary Investigator

Other Primary Investigator(s)

PI Name
Cha Li
PI Affiliation
University of Texas at Austin
Contact Investigator

Additional Trial Information

Status
In development
Start date
2025-05-19
End date
2026-09-01
Keywords
Firms & Productivity
Additional Keywords
JEL code(s)
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
We explore the role that AI may play in idea generation and evaluation processes.
External Link(s)

Registration Citation

Citation
Heshmati, Mana and Cha Li. 2025. "The Effect of AI on Idea Generation and Evaluation." AEA RCT Registry. May 21. https://doi.org/10.1257/rct.16016-1.0
Sponsors & Partners
Experimental Details

Interventions

Intervention(s)
See details below.
Intervention Start Date
2025-05-19
Intervention End Date
2025-05-30

Primary Outcomes

Primary Outcomes (end points)
See details below.
Primary Outcomes (explanation)
Outcomes of interest are idea generation ability and idea evaluation ability. For each participant we will measure the following outcomes:
(1) Idea generation task: Number of ideas generated, speed per idea generated, and quality of ideas
(2) Idea evaluation task: Extent to which business idea evaluation aligns with actual performance of the business ideas
(3) Cognitive role: The cognitive role individuals take during interactions with AI

We also plan to analyze the text of the conversation participants have with the AI tool (i.e., length of conversation, cognitive style, content), and the heterogeneity in treatment effects along the following dimensions: baseline evaluation skill, extent of regular AI usage, familiarity with industry, demographic characteristics.

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
Participants will complete the study across 3 stages. Participants will be randomly assigned to AI / non-AI use conditions. See details below for more information.
Experimental Design Details
Not available
Randomization Method
Randomly assigned via Qualtrics survey.
Randomization Unit
Individual level randomization.
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
200 participants
Sample size: planned number of observations
200 participants
Sample size (or number of clusters) by treatment arms
100 participants control group, 100 participants treatment group. These groups are further divided in half (50 each) across two different business "challenges" and are counterbalanced to avoid any ordering effects.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Supporting Documents and Materials
IRB

Institutional Review Boards (IRBs)

IRB Name
University of Washington Human Subjects Division (HSD)
IRB Approval Date
2025-04-02
IRB Approval Number
STUDY00022735
Analysis Plan