|
Field
Last Published
|
Before
November 19, 2024 04:38 PM
|
After
March 10, 2025 01:02 PM
|
|
Field
Intervention (Public)
|
Before
This study investigates how generative AI-modified descriptions of climate-related solutions impact the evaluation of their novelty, feasibility, and overall potential. The intervention involves creating three versions of each solution: a summary using the same wording and writing style as the original version, an AI-enhanced summary with novelty-promoting language, and an AI-neutralized summary where promotional language is stripped. Master students enrolled in a climate-focused course will evaluate the solutions, randomly receiving solutions from each of the three versions, and full original versions. The aim is to examine how AI-generated summarization and rephrasing affect the evaluators' perceptions and decisions. Each participant will evaluate 15 solutions selected from a pool of 150 versions (50 unique solutions, each in 3 different summary versions) and three original full solutions. Participants will also evaluate non-summarized versions to evaluate the summarization effect. There will be 100 non-summarized solutions (50 original submissions and 50 modified by AI). Each participant will evaluate 4 long solutions (2 originals and 2 modified by AI). For a total of 19 randomized evaluations.
|
After
This study investigates how generative AI-modified descriptions of climate-related solutions impact the evaluation of their novelty, feasibility, and overall potential. The intervention involves creating three versions of each solution: a summary using the same wording and writing style as the original version, an AI-enhanced summary with novelty-promoting language, and an AI-neutralized summary where promotional language is stripped. Master students enrolled in a climate-focused course will evaluate the solutions, randomly receiving solutions from each of the three versions, and full original versions. The aim is to examine how AI-generated summarization and rephrasing affect the evaluators' perceptions and decisions. Each participant will evaluate 15 solutions selected from a pool of 150 versions (50 unique solutions, each in 3 different summary versions) and three original full solutions. Participants will also evaluate non-summarized versions to evaluate the summarization effect. There will be 100 non-summarized solutions (50 original submissions and 50 modified by AI). Each participant will evaluate 4 long solutions (2 originals and 2 modified by AI). For a total of 19 randomized evaluations.
We run a second-round of experimentation in the same empirical context with a simplified version with only two treatments in addition to the control. The treatment consist in two summarized and standardized versions of the control using AI. One long and one short.
|
|
Field
Experimental Design (Public)
|
Before
This randomized controlled trial (RCT) tests how generative AI-modified descriptions affect the evaluation of climate-related solutions. 100 Master students in two separate institutions will evaluate 19 solutions, randomly selected from 250 versions (50 solutions in 5 variations). Each participant will receive 5 solutions per summarized version type (original summarized, AI-enhanced summarized, AI-neutralized summarized), with the order randomized, and four long versions. The study aims to uncover the effects of AI-modified framing on perceptions of novelty and feasibility.
|
After
This randomized controlled trial (RCT) tests how generative AI-modified descriptions affect the evaluation of climate-related solutions. 100 Master students in two separate institutions will evaluate 19 solutions, randomly selected from 250 versions (50 solutions in 5 variations). Each participant will receive 5 solutions per summarized version type (original summarized, AI-enhanced summarized, AI-neutralized summarized), with the order randomized, and four long versions. The study aims to uncover the effects of AI-modified framing on perceptions of novelty and feasibility.
The second round will consist of 12 solutions to evaluate from 180 versions (60 solutions in 3 conditions).
|
|
Field
Planned Number of Clusters
|
Before
2
|
After
2
For the second round. Two schools as well as a group of experts in sustainability.
|
|
Field
Sample size (or number of clusters) by treatment arms
|
Before
Each participant will evaluate 19 solutions, distributed across the three treatment arms and the original full version:
Original summary version: ~500 evaluations
AI-enhanced summary version: ~500 evaluations
AI-neutralized summary version: ~500 evaluations
Original full version: ~200 evaluations
Modified full version: ~200 evaluations
|
After
Each participant will evaluate 19 solutions, distributed across the three treatment arms and the original full version:
Original summary version: ~500 evaluations
AI-enhanced summary version: ~500 evaluations
AI-neutralized summary version: ~500 evaluations
Original full version: ~200 evaluations
Modified full version: ~200 evaluations
Second round:
Original version: ~240 observations (4 solutions *60 participants)
AI-summarized-long: ~240 observations (4 solutions *60 participants)
AI-summarized-short: ~240 observations (4 solutions *60 participants)
|
|
Field
Secondary Outcomes (Explanation)
|
Before
Participants will also be asked to judge the:
Clarity about solution: After reading the description of the solution above, you have a clear idea of what the solution is about.
Ease of readability: How easy was the process of reading the solution description. Bias Detection: Measured through changes in novelty and feasibility ratings across different description types. We will analyze whether evaluators are more biased toward either novelty-enhanced or neutralized descriptions.
Consistency in Evaluation: By comparing evaluations across multiple evaluators, we measure the standard deviation of novelty and feasibility scores, hypothesizing that AI-generated summaries might produce more consistent evaluations.
AI-Human Comparison: We will compare human evaluators' novelty ratings with those derived from AI-based text embedding models to identify disparities between subjective human and objective AI assessments.
|
After
Participants will also be asked to judge the:
Clarity about solution: After reading the description of the solution above, you have a clear idea of what the solution is about.
Ease of readability: How easy was the process of reading the solution description. Bias Detection: Measured through changes in novelty and feasibility ratings across different description types. We will analyze whether evaluators are more biased toward either novelty-enhanced or neutralized descriptions.
Consistency in Evaluation: By comparing evaluations across multiple evaluators, we measure the standard deviation of novelty and feasibility scores, hypothesizing that AI-generated summaries might produce more consistent evaluations.
AI-Human Comparison: We will compare human evaluators' novelty ratings with those derived from AI-based text embedding models to identify disparities between subjective human and objective AI assessments.
Confidence in answer.
|