Experimental Design Details
Experimental Design
Procedure The experiment will be fully computerized using Qualtrics and will be conducted online (e.g., Prolific). The study will begin with subjects signing the informed consent form. Consenting subjects will be taken to a welcome screen explaining the general structure of the experiment. The study is comprised of two parts of decision-making followed by a brief post-experiment questionnaire. Subjects will face attention checks through the survey. After the survey, subjects will be invited to read a debriefing statement. We anticipate that the study will last on average about 10 minutes. Sample instructions are provided as part of this pre-registration.
Payments All participants will be paid a participation fee of $3 . In addition, each of the two main parts of the survey will provide the subjects an opportunity to earn bonus payments depending on their decisions. Extra earnings will vary based on your answers in the study. Participants who successfully complete the survey are expected to earn around $5 in total. Incentives are based on this estimate in order to ensure an hourly wage of $12/hour.
IRB Approval The study has obtained approval from the Institutional Review Board of Tufts University (Tufts IRB STUDY00005291), with a reliance agreement extended to Wellesley College under Brandeis University’s Human Research Protection Program.
Experiment and Treatments Participants will be informed that their task in the study would entail providing assessments of public statements sourced from online sources. In Part 1, subjects will read seven different statements and, for each, select their preferred response from a list of multiple (4) options.
Prior to Part 1, subjects will be randomized at the individual level into four treatment conditions described in the table below:
No Oath; Private Response (NoP) Oath; Private Response (OP)
No Oath; Broadcast Response (NoB) Oath; Broadcast Response (OB)
Subjects randomized into the Oath Treatments (OB and OP) will first read and type in the paragraph in Italics below which represents a pledge to act with civility. Typing this passage correctly will provide the subject with a $1 bonus payment. The purpose of this step is to ensure that subjects pay attention to the content of the pledge. In order to prevent subjects from simply copying and pasting the pledge into the box without paying attention to its content, we will fix the minimum time they must spend on this screen to 30 seconds and remind them not to copy/paste.
Oath: I pledge to promote civility and respect in all my online interactions. I will not use offensive language or attack others based on their race, gender, or socioeconomic level. I will strive to engage in constructive dialogue, listening with empathy, and responding with kindness.
Subjects randomized into the Control Treatments without an oath (NoB and NoP) will also type in a passage to ensure that any effects found are not due purely to the act of typing. Typing the control passage correctly will provide the subject with a $1 bonus payment.
Control passage: The internet began evolving in the 1960s and became publicly available in the early 1990s. It has transformed global communication and introduced new considerations such as privacy, reflecting its complex role in society.
Following the reading and typing screen, subjects will view another information screen with an explanation of the next task. They will learn that they will view a series of posts (which we call “statements”) from actual online sources and will be asked to provide a response to each by choosing from a pre-specified list. These responses should best match the ones that they would use to respond to the same statement online in a public forum.
Subjects randomized into the public response treatments (NoP and OP) will also learn that we will randomly select a set of responses from the submissions and anonymously post them in comments on these public statements. They will be asked a comprehension question to make sure they understand if they have been randomized into the treatment where their responses may be publicly posted (or not). If they answer this question incorrectly we will make them aware of the correct answer. This means that the response(s) could be observed publicly by anyone who is reading the comments online. (We will create an account that we will use to post the selected responses after the experiment has concluded.)
The list of statements has been sourced from Politifact and confirmed to exist on an online forum which is open to public comments. The responses were generated by ChatGPT version 4o (the latest and most advanced model) using the following prompt:
Please provide potential agreeing and disagreeing reactions to the statement in quotes, ranging in terms of vehemence and civility with which the reaction is phrased, basing these reactions on typical social media comments and replies on similar posts. “Joe Biden is letting millions of people from jails, from prisons, from insane asylums, from mental institutions, drug dealers pour in.” (Source: Donald Trump; stated on May 29, 2024 in comments to reporters, https://www.politifact.com/factchecks/2024/jun/06/donald-trump/fact-check-trumps-ridiculous-claim-that-millions-o/)
Note that the statement above is an example, and we will not use it in the experiment. ChatGPT will provide several agreeing and disagreeing responses, ranging along the spectrum of civility. For example, for the statement above, the four selected responses would be:
BIDEN’S A DISASTER!!! He's opening doors to all kinds of scum. We're not safe anymore!!! (Agreeing; Uncivil)
I agree. Biden's policies are endangering our society by allowing too many dangerous people in. (Agreeing; Civil)
This claim is completely unfounded and dangerous. Biden is not letting criminals run wild. (Disagreeing; Civil)
This is NONSENSE!!! completely unfounded and dangerous!!! This is fear-mongering at its worst! (Disagreeing; Uncivil)
Lack of civility will be indicated by use of all capital letters and exclamation marks, as well as overall tone being more antagonistic and hostile. In some responses, we will include hashtags to enhance ecological validity of an online forum setting. We will refrain from using any extremely abusive or offensive language to remain within the bounds of ethics (all statements have been reviewed and approved by the IRB). The broad topics for each statement and the information on whether the statement is false, true, or half-true will be sourced from Politifact’s fact-checking platform.
The subjects will view and choose responses to seven statements, presented to them in random order. Each statement will appear on its own screen with a list of responses. The order of responses will also be randomized at the participant level.
In Part 2, participants will provide their beliefs about the same set of seven statements, presented in random order. For each statement, participants will first indicate whether they believe it is True, False, or Half-True. On separate screens that follow, they will be asked to share their opinion on the political ideology associated with the statement and to assess whether the statement was more likely made by a man, a woman, or neither. These questions will be implemented as sliders that subjects can move on a scale ranging from very liberal (0) to very conservative (100) and male (0) to female (100), respectively. Lastly participants will be prompted to guess the author of each statement by typing a name into a text box.
To incentivize effort, we will pick one out of seven statements to count for bonus. For that randomly chosen statement, participants will be rewarded $0.50 if their assessment of its veracity aligns with Politifact’s fact-checking result. Similarly, for the author guess, $0.50 will be awarded if the guess is correct (minor spelling errors will not result in penalties, but both first and last names will be required).
To reduce the likelihood that participants will search online for the source of the statements, we will monitor whether they navigate away from the survey window, which would indicate they may be using other websites to obtain information. Additionally, a time limit of 45 seconds will be imposed on each response to limit the ability of subjects to use another device to look up answers during the survey.
Participants will face attention checks throughout. In Part 1, we will add an extra (eighth) statement with four responses, but will ask them simply to select a certain prespecified reaction (e.g., reaction B) from the list on the screen. In Part 2, participants will be asked to provide a particular answer (e.g., Please select Half-True/Half-False for this question; Please select between 70 and 80 for this question regardless of your actual opinion). We will not remove participants who fail the attention checks, but we will perform robustness checks without these subjects in the analysis.
After responding to all the statements and providing their perceptions about the statements, participants will move on to the post-experiment questionnaire. Here we will ask them to provide demographics, as well as survey questions eliciting their political ideology, risk preferences, and social preferences. In order to gauge subjects’ understanding of the experiment, we will also ask them to provide us with free response feedback, including their perceptions of what they thought the study was about.