Evaluating the impact of AI-powered reasoning in deliberative processes

We test two main interventions for improving argument quality in an online policy debate on mental illness and firearm regulation in the United States:

1) AI-powered Reasoning: Participants receive prompts from a large language model (LLM) designed to foster more reflective, logically coherent arguments (i.e., Socratic dialogue), solo guided reflection (i.e., Reflective paragraph); LLM-powered emotional regulation (i.e., Emotional-regulation LLM); grammar correction (i.e., Grammar-correction LLM), or waiting control.

2) Monetary Incentives: Participants receive the option to receive direct monetary bonus (for top quartile arguments), a lottery-based bonus (for top quartile arguments), or no incentive.

2025-02-06

2025-02-28

1. Argument thoughtfulness score: A composite measure (0–6 scale) reflecting logical structure, clarity, and depth of each participant’s main comment.

2. Position shifting or compromise: The extent to which participants move from an extreme stance toward a middle-ground position on the policy.

1. Argument Thoughtfulness Score reflecting three dimensions (clarity, coherence, and depth), each rated by independent coders (Human, LLM, and index of both).

2. Position shifting is computed as the absolute change in stance on a -5 to +5 slider (e.g., from strongly oppose to moderately oppose). Additionally, we consider endorsement of seeded comments and a policy proposal, as well as willingness to donate to a non-profit organization.

1. Cognitive Reflection: Changes in CRT scores (pre-post).
2. Emotional State: Pre-post changes on a 2-dimension affect scale.
3. Perspective Diversity: Type-token ratio, semantic similarity variance, voting weighted average.
4. Engagement Indices: Number of comments edited or length of comment, voting behavior, willingness to donate, time spent.

1. Cognitive Reflection is measured via a standard 2 CRT questions, taking the difference between baseline and endline.
2. Emotional State is captured using self-reported activation and pleasure scales (0–10).
3. Perspective Diversity in textual output is assessed via NLP metrics such as TTR (type-token ratio) and semantic embedding variance. Also, we estimate voting weighted average, considering the participant's self-reported position and agreement on seeded comments.
4. Engagement includes votes on others’ comments, revisions made, willingness to donate (binary), and time spent.

We conduct an online experiment with participants recruited to debate whether people with mental illnesses should be allowed to purchase firearms. We randomize at the individual level along two dimensions:
(i) reasoning interventions (Socratic dialogue LLM vs. Reflective paragraph vs. Emotional-regulation LLM vs. Grammar-correction LLM vs. waiting control) and
(ii) monetary incentives (direct bonus vs. lottery vs. none).

The primary objective is to determine which intervention(s) most effectively increase argument thoughtfulness and willingness to compromise. We measure outcomes post-treatment via rubric-based coding of each participant’s main argument, changes in stance, and secondary indicators like voting and donation behavior.

Final analyses will compare each treatment arm with pooled or disaggregated controls, depending on tests of equivalence among control conditions.

Details in Pre-Analysis Plan attached.

Not available

Pseudo-random generator within the survey/intervention platform (deliberation.io). Each participant has equal probability of assignment to each cell in the 5×3 factorial design.

Individual.

No

N/A.

1,500 when analyzing individual-level outcomes. Up to 18,000 (1500 x 12) when analyzing individual-vote level outcomes.

1500: 300 for each reasoning condition; 500 for each monetary incentive condition.

1. Argument Thoughtfulness score: Pilot data (non-registered pilot in September 2024) suggests a SD ≈ 2.28 on a 0–6 scale. With n=1,500, we can detect a mean difference of roughly 0.3–0.55 points (on 6-point scale) with 80% power. 2. Position Shift (Compromise): Pilot data (non-registered pilot in September 2024) suggests a SD ≈ 0.458 on a binary measure. With n=1,500, we can detect a mean difference of roughly 6-10 percentage points with 80% power.