BallotBot: Can Chatbots Strengthen Direct Democracy?

Last registered on November 14, 2024
View Trial History

Pre-Trial

Trial Information

General Information

Title
BallotBot: Can Chatbots Strengthen Direct Democracy?
RCT ID
AEARCTR-0014172
Initial registration date
October 10, 2024

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
October 18, 2024, 4:50 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
November 14, 2024, 1:22 PM EST

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Country
United States of America
Region
California

Primary Investigator

Name
Giacomo Opocher
Affiliation
University of Bologna
Contact Primary Investigator

Other Primary Investigator(s)

PI Name
Elliott Ash
PI Affiliation
ETH - Zurich
PI Name
Sergio Galletta
PI Affiliation
ETH - Zurich

Additional Trial Information

Status
Completed
Start date
2024-10-11
End date
2024-11-14
Keywords
Electoral
Additional Keywords
Artificial Intelligence, Political Knowledge
JEL code(s)
D72, D83, L82
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
In this study, we investigate the impact of new information technologies on American voters' political knowledge through a two-wave online survey experiment. We developed BallotBot, an interactive chatbot powered by state-of-the-art large language models (LLMs) trained to provide official information about statewide ballot measures. A random sample of California voters is assigned to either use BallotBot or the digital version of the official voter guide to answer questions about upcoming ballot initiatives. We want to compare participants' short- and medium-term political knowledge, as well as the perceived and objective costs of acquiring that knowledge, across the two experimental groups. Additionally, we collect data on voting intentions, turnout behavior, and the reasoning behind participants' vote choices to assess how these factors differ between the groups.
External Link(s)

Registration Citation

Citation
Ash, Elliott, Sergio Galletta and Giacomo Opocher. 2024. "BallotBot: Can Chatbots Strengthen Direct Democracy?." AEA RCT Registry. November 14. https://doi.org/10.1257/rct.14172-1.1
Sponsors & Partners
Experimental Details

Interventions

Intervention(s)
Intervention (Hidden)
Intervention Start Date
2024-10-11
Intervention End Date
2024-10-18

Primary Outcomes

Primary Outcomes (end points)
1. Political Knowledge: number of correct answers to the quiz.
2. Objective Cost of Producing Knowledge: time needed to complete each question.
3. Subjective Cost of Producing Knowledge: bid selected at the BDM task.
4. Turnout Behaviour.
*details in the Pre Analysis Plan
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
The experiment takes the form of a survey where participants receive an information treatment and are asked to answer a few questions. Participants' responses are collected via Qualtrics, and they are recruited through Prolific.

We will first collect baseline characteristics of all respondents. After that, we will split the participants into two groups:
T1 Group: This group will receive the digital version of the voter guide that the government distributes.
T2 Group: This group will access a simple chatbot (in the form of a web application) that exclusively provides official information from the voter guide and can only answer questions related to the referendum.

Then, we will ask the participants to answer two sets of questions:
– 5 compulsory closed questions on the Propositions’ content. To answer correctly, they can use the chatbot (T2) or the guide (T1), depending on what they are assigned to. We will measure (i) the number of correct answers (knowledge) and (ii) the time they need to answer (objective cost of generating knowledge).
– 1 voluntary closed question on the Propositions’ content. We will ask them what fee (compensation for a correct answer) they would be willing to accept for answering each question, in the style of a BDM task. We will collect (i) the answer, (ii) the time they need to answer, and (iii) the bid (perceived cost of generating knowledge).
– We will encourage them to voluntarily write a question of their interest and look for the answer using the device they were assigned to.
– Finally, we will ask a last set of questions about their willingness to use BallotBot in future elections.

One week later, we will contact the participants again and:
– Repeat some of the quiz questions to measure retained knowledge.
– Ask about their turnout behavior.
– Ask them to write a short text to motivate their vote choice.

See the PAP for further details.
Experimental Design Details
Randomization Method
Automatically by Qualtrics
Randomization Unit
Individual
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
0
Sample size: planned number of observations
2400 individuals
Sample size (or number of clusters) by treatment arms
2400 individuals
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Supporting Documents and Materials

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
IRB

Institutional Review Boards (IRBs)

IRB Name
Ethical Committee of the University of Bologna
IRB Approval Date
2024-07-24
IRB Approval Number
0217279
Analysis Plan

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
Yes
Intervention Completion Date
October 18, 2024, 12:00 +00:00
Data Collection Complete
Yes
Data Collection Completion Date
November 14, 2024, 12:00 +00:00
Final Sample Size: Number of Clusters (Unit of Randomization)
/
Was attrition correlated with treatment status?
No
Final Sample Size: Total Number of Observations
1464
Final Sample Size (or Number of Clusters) by Treatment Arms
732 and 732
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
No
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials