Reframing Innovation with Generative AI: The Impact of Modified Descriptions on Idea Evaluation

Last registered on November 19, 2024

View Trial History

Pre-Trial

Trial Information

General Information

Title

Reframing Innovation with Generative AI: The Impact of Modified Descriptions on Idea Evaluation

RCT ID

AEARCTR-0014529

Initial registration date

November 18, 2024

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published

November 19, 2024, 4:38 PM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Primary Investigator

Name

Charles Ayoubi

Affiliation

ESSEC Business School

Contact Primary Investigator

Other Primary Investigator(s)

PI Name

Julian Just

PI Affiliation

UIBK

Contact Investigator

Additional Trial Information

Status

In development

Start date

2024-11-19

End date

2025-11-30

Keywords

Behavior, Firms & Productivity

Additional Keywords

JEL code(s)

O31, O32, C93, D8

Secondary IDs

Prior work

This trial does not extend or rely on any prior RCTs.

Abstract

This study emphasizes the role of generative AI in modifying the descriptions of innovative solutions and how these modifications influence evaluation outcomes of humans and of AI. Specifically, the research explores how summarization, which condenses solutions into more essential and streamlined versions, and targeted reframing, which changes the writing style and wording for desired purposes, affects human evaluators' perceptions of novelty, feasibility, and other evaluation metrics.
The study's inquiry revolves around AI's ability to reframe ideas by either enhancing or neutralizing the novelty presented in these descriptions. By creating AI-modified summaries, the experiment tests whether these modifications change evaluators' views on the novelty and feasibility of early-stage ideas. The AI-neutralized descriptions allow the team to control for cognitive biases that may arise due to detailed or promotional language. Thus, the study will also investigate differences between human and AI evaluation depending on the summarization and framing.

Key points of consideration include:
Summarization and Cognitive Load: Simplified AI-generated summaries reduce the cognitive burden on evaluators, potentially leading to less biased and more objective assessments.
Neutralization vs. Promotional Framing: The study compares the effects of AI-neutralized descriptions, where promotional language is stripped, versus AI-enhanced versions that promote novelty. It investigates whether these framings impact novelty perception and decision-making differently.
Perceived vs. Objective Novelty: AI-modified descriptions allow a direct comparison between human-perceived novelty, AI-perceived novelty (prompted GPT model), and objective novelty measures (similarity embedding techniques). The research questions whether AI summarization can bridge the gap between these dimensions.

This study thus aims to shed light on how AI-driven text modifications can influence innovation evaluation, providing insights into the role of linguistic framing and summarization in decision-making processes.

External Link(s)

Registration Citation

Citation

Ayoubi, Charles and Julian Just. 2024. "Reframing Innovation with Generative AI: The Impact of Modified Descriptions on Idea Evaluation." AEA RCT Registry. November 19. https://doi.org/10.1257/rct.14529-1.0

Sponsors & Partners

Experimental Details

Interventions

Intervention(s)

This study investigates how generative AI-modified descriptions of climate-related solutions impact the evaluation of their novelty, feasibility, and overall potential. The intervention involves creating three versions of each solution: a summary using the same wording and writing style as the original version, an AI-enhanced summary with novelty-promoting language, and an AI-neutralized summary where promotional language is stripped. Master students enrolled in a climate-focused course will evaluate the solutions, randomly receiving solutions from each of the three versions, and full original versions. The aim is to examine how AI-generated summarization and rephrasing affect the evaluators' perceptions and decisions. Each participant will evaluate 15 solutions selected from a pool of 150 versions (50 unique solutions, each in 3 different summary versions) and three original full solutions. Participants will also evaluate non-summarized versions to evaluate the summarization effect. There will be 100 non-summarized solutions (50 original submissions and 50 modified by AI). Each participant will evaluate 4 long solutions (2 originals and 2 modified by AI). For a total of 19 randomized evaluations.

Intervention Start Date

2024-11-19

Intervention End Date

2025-06-27

Primary Outcomes

Primary Outcomes (end points)

Potential for Impact: The planned solution implementation has the potential to impact the intended population.
Feasibility: The solution suggests a realistic, practical plan for being implemented, and it is feasible in the given context.
Innovative Approach: The solution includes a new technology, a new application of technology, a new business model, or a new process for solving the Challenge.

Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)

Bias Detection: Whether summarization and/or AI-modified descriptions (enhanced or neutralized) increase or decrease evaluators’ biases toward novelty or feasibility.
Participants will also be asked to judge the:
Clarity about solution: After reading the description of the solution above, you have a clear idea of what the solution is about.
Ease of readability: How easy was the process of reading the solution description.
Consistency in Evaluation: The variance in evaluators' responses when exposed to AI-modified vs. original descriptions.
Comparison with AI Evaluation: How human evaluation of novelty compares to the novelty scores generated by the AI system.

Secondary Outcomes (explanation)

Participants will also be asked to judge the:
Clarity about solution: After reading the description of the solution above, you have a clear idea of what the solution is about.
Ease of readability: How easy was the process of reading the solution description. Bias Detection: Measured through changes in novelty and feasibility ratings across different description types. We will analyze whether evaluators are more biased toward either novelty-enhanced or neutralized descriptions.
Consistency in Evaluation: By comparing evaluations across multiple evaluators, we measure the standard deviation of novelty and feasibility scores, hypothesizing that AI-generated summaries might produce more consistent evaluations.
AI-Human Comparison: We will compare human evaluators' novelty ratings with those derived from AI-based text embedding models to identify disparities between subjective human and objective AI assessments.

Experimental Design

This randomized controlled trial (RCT) tests how generative AI-modified descriptions affect the evaluation of climate-related solutions. 100 Master students in two separate institutions will evaluate 19 solutions, randomly selected from 250 versions (50 solutions in 5 variations). Each participant will receive 5 solutions per summarized version type (original summarized, AI-enhanced summarized, AI-neutralized summarized), with the order randomized, and four long versions. The study aims to uncover the effects of AI-modified framing on perceptions of novelty and feasibility.

Experimental Design Details

Not available

Randomization Method

Randomization will be done by a computer algorithm from Qualtrics that assigns each participant to a random selection of solutions and randomizes the order in which the solutions are presented.

Randomization Unit

The unit of randomization is at the individual level, where each participant receives 19 randomly selected solutions in randomized order across the three treatment types (original summary, AI-enhanced summary, and AI-neutralized summary) and the four long full versions. There is no clustering.

Was the treatment clustered?

Experiment Characteristics

Sample size: planned number of clusters

Sample size: planned number of observations

The planned number of observations is 1900 evaluations (100 students x 19 evaluations per student).

Sample size (or number of clusters) by treatment arms

Each participant will evaluate 19 solutions, distributed across the three treatment arms and the original full version:
Original summary version: ~500 evaluations
AI-enhanced summary version: ~500 evaluations
AI-neutralized summary version: ~500 evaluations
Original full version: ~200 evaluations
Modified full version: ~200 evaluations

Minimum detectable effect size for main outcomes (accounting for sample design and clustering)

The minimum detectable effect size (MDES) is estimated at 0.4 standard deviations, based on a sample of 1500 evaluations, accounting for within-subject variability and aiming for a power of 0.8 at a 5% significance level. This estimate is based on expected variance in perceived novelty and feasibility scores across the three treatment conditions.

Supporting Documents and Materials

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

IRB

Institutional Review Boards (IRBs)

IRB Name

Board for Ethical Questions in Science of the University of Innsbruck

IRB Approval Date

2024-11-12

IRB Approval Number

89/2024

Analysis Plan