Reframing Innovation with Generative AI: The Impact of Modified Descriptions on Idea Evaluation

Last registered on March 10, 2025

View Trial History

Pre-Trial

Trial Information

General Information

Title

Reframing Innovation with Generative AI: The Impact of Modified Descriptions on Idea Evaluation

RCT ID

AEARCTR-0014529

Initial registration date

November 18, 2024

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published

November 19, 2024, 4:38 PM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated

March 10, 2025, 1:02 PM EDT

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Country

Region

Country

Region

Primary Investigator

Name

Charles Ayoubi

Affiliation

ESSEC Business School

Contact Primary Investigator

Other Primary Investigator(s)

PI Name

Julian Just

PI Affiliation

UIBK

Contact Investigator

Additional Trial Information

Status

In development

Start date

2024-11-19

End date

2025-11-30

Keywords

Behavior, Firms & Productivity

Additional Keywords

JEL code(s)

O31, O32, C93, D8

Secondary IDs

Prior work

This trial does not extend or rely on any prior RCTs.

Abstract

This study emphasizes the role of generative AI in modifying the descriptions of innovative solutions and how these modifications influence evaluation outcomes of humans and of AI. Specifically, the research explores how summarization, which condenses solutions into more essential and streamlined versions, and targeted reframing, which changes the writing style and wording for desired purposes, affects human evaluators' perceptions of novelty, feasibility, and other evaluation metrics.
The study's inquiry revolves around AI's ability to reframe ideas by either enhancing or neutralizing the novelty presented in these descriptions. By creating AI-modified summaries, the experiment tests whether these modifications change evaluators' views on the novelty and feasibility of early-stage ideas. The AI-neutralized descriptions allow the team to control for cognitive biases that may arise due to detailed or promotional language. Thus, the study will also investigate differences between human and AI evaluation depending on the summarization and framing.

Key points of consideration include:
Summarization and Cognitive Load: Simplified AI-generated summaries reduce the cognitive burden on evaluators, potentially leading to less biased and more objective assessments.
Neutralization vs. Promotional Framing: The study compares the effects of AI-neutralized descriptions, where promotional language is stripped, versus AI-enhanced versions that promote novelty. It investigates whether these framings impact novelty perception and decision-making differently.
Perceived vs. Objective Novelty: AI-modified descriptions allow a direct comparison between human-perceived novelty, AI-perceived novelty (prompted GPT model), and objective novelty measures (similarity embedding techniques). The research questions whether AI summarization can bridge the gap between these dimensions.

This study thus aims to shed light on how AI-driven text modifications can influence innovation evaluation, providing insights into the role of linguistic framing and summarization in decision-making processes.

External Link(s)

Registration Citation

Citation

Ayoubi, Charles and Julian Just. 2025. "Reframing Innovation with Generative AI: The Impact of Modified Descriptions on Idea Evaluation." AEA RCT Registry. March 10. https://doi.org/10.1257/rct.14529-1.1

Sponsors & Partners

Experimental Details

Interventions

Intervention(s)

This study investigates how generative AI-modified descriptions of climate-related solutions impact the evaluation of their novelty, feasibility, and overall potential. The intervention involves creating three versions of each solution: a summary using the same wording and writing style as the original version, an AI-enhanced summary with novelty-promoting language, and an AI-neutralized summary where promotional language is stripped. Master students enrolled in a climate-focused course will evaluate the solutions, randomly receiving solutions from each of the three versions, and full original versions. The aim is to examine how AI-generated summarization and rephrasing affect the evaluators' perceptions and decisions. Each participant will evaluate 15 solutions selected from a pool of 150 versions (50 unique solutions, each in 3 different summary versions) and three original full solutions. Participants will also evaluate non-summarized versions to evaluate the summarization effect. There will be 100 non-summarized solutions (50 original submissions and 50 modified by AI). Each participant will evaluate 4 long solutions (2 originals and 2 modified by AI). For a total of 19 randomized evaluations.

We run a second-round of experimentation in the same empirical context with a simplified version with only two treatments in addition to the control. The treatment consist in two summarized and standardized versions of the control using AI. One long and one short.

Intervention (Hidden)

The randomized controlled trial (RCT) involves presenting around 100 Master students with three types of descriptions for each climate-related solution: the original summary version, an enhanced summary version (which increases the promotional novelty framing using generative AI), and a neutralized summary version (which removes promotional or emotional language). Evaluators will be randomly assigned 15 solutions in each of these three versions and three original full versions of the solutions, and their evaluations will be analyzed to assess the effects of AI modifications on perceived novelty, feasibility, and other biases. Both human and AI-generated evaluations will be compared to analyze discrepancies in evaluation outcomes. Participants will also evaluate non-summarized versions to evaluate the summarization effect. There will be 100 non-summarized solutions (50 original submissions and 50 modified by AI). Each participant will evaluate 4 long solutions (2 originals and 2 modified by AI). For a total of 19 randomized evaluations.

The trial involves randomizing around 100 Master students to evaluate 19 climate-related solutions. These solutions are randomly drawn from 250 versions (50 unique solutions, each presented in five formats: original, original modified, original-summarized, AI-enhanced, and AI-neutralized). Each participant receives five solutions from each summarized version type (original-summarized, enhanced, neutralized) and four long versions (2 original and 2 modified), with the order of presentation fully randomized. This randomized controlled trial (RCT) will allow us to measure the impact of generative AI-based summarization and framing on perceived novelty, feasibility, and bias in evaluations. Additionally, AI-generated novelty and feasibility assessments will be compared to human evaluations to investigate potential disparities.

Intervention Start Date

2024-11-19

Intervention End Date

2025-06-27

Primary Outcomes

Primary Outcomes (end points)

Potential for Impact: The planned solution implementation has the potential to impact the intended population.
Feasibility: The solution suggests a realistic, practical plan for being implemented, and it is feasible in the given context.
Innovative Approach: The solution includes a new technology, a new application of technology, a new business model, or a new process for solving the Challenge.

Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)

Bias Detection: Whether summarization and/or AI-modified descriptions (enhanced or neutralized) increase or decrease evaluators’ biases toward novelty or feasibility.
Participants will also be asked to judge the:
Clarity about solution: After reading the description of the solution above, you have a clear idea of what the solution is about.
Ease of readability: How easy was the process of reading the solution description.
Consistency in Evaluation: The variance in evaluators' responses when exposed to AI-modified vs. original descriptions.
Comparison with AI Evaluation: How human evaluation of novelty compares to the novelty scores generated by the AI system.

Secondary Outcomes (explanation)

Participants will also be asked to judge the:
Clarity about solution: After reading the description of the solution above, you have a clear idea of what the solution is about.
Ease of readability: How easy was the process of reading the solution description. Bias Detection: Measured through changes in novelty and feasibility ratings across different description types. We will analyze whether evaluators are more biased toward either novelty-enhanced or neutralized descriptions.
Consistency in Evaluation: By comparing evaluations across multiple evaluators, we measure the standard deviation of novelty and feasibility scores, hypothesizing that AI-generated summaries might produce more consistent evaluations.
AI-Human Comparison: We will compare human evaluators' novelty ratings with those derived from AI-based text embedding models to identify disparities between subjective human and objective AI assessments.

Confidence in answer.

Experimental Design

This randomized controlled trial (RCT) tests how generative AI-modified descriptions affect the evaluation of climate-related solutions. 100 Master students in two separate institutions will evaluate 19 solutions, randomly selected from 250 versions (50 solutions in 5 variations). Each participant will receive 5 solutions per summarized version type (original summarized, AI-enhanced summarized, AI-neutralized summarized), with the order randomized, and four long versions. The study aims to uncover the effects of AI-modified framing on perceptions of novelty and feasibility.

The second round will consist of 12 solutions to evaluate from 180 versions (60 solutions in 3 conditions).

Experimental Design Details

The study utilizes a within-subjects design, with participants randomly assigned to all of the three treatment groups: 1) evaluating the original solution description, 2) evaluating the summary of the original solution description, 3) evaluating an AI-enhanced summary version that promotes novelty, or 4) evaluating an AI-neutralized summary version that removes promotional or emotional language. Participants will evaluate 19 solutions selected randomly from a pool of 250 versions (50 unique solutions, each presented in five different forms). Each participant will receive about five solutions in each summarized format (original summary, AI-enhanced summary, and AI-neutralized summary) and four in the long format, and the order in which these are presented will be fully randomized. Afterwards, three original full versions will be evaluated by each participant based on randomized assignment.
Evaluations are measured across several dimensions, including perceived novelty, feasibility, and other overall evaluation scores. The goal is to isolate the effects of f different framing types and the summarization on evaluators' perceptions and decision-making processes. Additionally, AI-generated evaluations (based on prompted GPT models and based on semantic distance measures) of the same solutions will be compared to human assessments. This may allow us to identify potential biases and inconsistencies.

Randomization Method

Randomization will be done by a computer algorithm from Qualtrics that assigns each participant to a random selection of solutions and randomizes the order in which the solutions are presented.

Randomization Unit

The unit of randomization is at the individual level, where each participant receives 19 randomly selected solutions in randomized order across the three treatment types (original summary, AI-enhanced summary, and AI-neutralized summary) and the four long full versions. There is no clustering.

Was the treatment clustered?

Experiment Characteristics

Sample size: planned number of clusters

2

For the second round. Two schools as well as a group of experts in sustainability.

Sample size: planned number of observations

The planned number of observations is 1900 evaluations (100 students x 19 evaluations per student).

Sample size (or number of clusters) by treatment arms

Each participant will evaluate 19 solutions, distributed across the three treatment arms and the original full version:
Original summary version: ~500 evaluations
AI-enhanced summary version: ~500 evaluations
AI-neutralized summary version: ~500 evaluations
Original full version: ~200 evaluations
Modified full version: ~200 evaluations

Second round:
Original version: ~240 observations (4 solutions *60 participants)
AI-summarized-long: ~240 observations (4 solutions *60 participants)
AI-summarized-short: ~240 observations (4 solutions *60 participants)

Minimum detectable effect size for main outcomes (accounting for sample design and clustering)

The minimum detectable effect size (MDES) is estimated at 0.4 standard deviations, based on a sample of 1500 evaluations, accounting for within-subject variability and aiming for a power of 0.8 at a 5% significance level. This estimate is based on expected variance in perceived novelty and feasibility scores across the three treatment conditions.

Supporting Documents and Materials

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

IRB