Back to History Current Version

Predicting Biased Polls

Last registered on October 04, 2023
View Trial History

Pre-Trial

Trial Information

General Information

Title
Predicting Biased Polls
RCT ID
AEARCTR-0005186
Initial registration date
February 11, 2020

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
February 12, 2020, 1:40 PM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
October 04, 2023, 11:48 AM EDT

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Country
United States of America
Region

Primary Investigator

Name
Andy Brownback
Affiliation
University of Arkansas
Contact Primary Investigator

Other Primary Investigator(s)

PI Name
Tristan Gagnon-Bartsch
PI Affiliation
Harvard University
Contact Investigator
PI Name
Nathaniel Burke
PI Affiliation
University of Arkansas
Contact Investigator

Additional Trial Information

Status
Completed
Start date
2020-02-11
End date
2020-06-01
Keywords
Electoral, Governance
Additional Keywords
Polling, Social Desirability, Beliefs
JEL code(s)
D9, D7
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
Unincentivized surveys of stated-preference may be susceptible to "socially desirable responding" (SDR, hereafter) where respondents strategically conceal preferences that they perceive to be socially undesirable. In contrast, economists often assume that the incentive-compatibility of revealed-preference elicitations mitigates the threat of SDR by imposing real costs. One domain in which SDR appears particularly powerful is the realm of politics where SDR is both ubiquitous and well-known by the popular press and other "consumers" of polling research. Moreover, incentive-compatible elicitations of political preferences are not feasible. However, if awareness of SDR is sufficient to de-bias polling information, then SDR may not cause any negative consequences. This project seeks to test the ability of economic agents to de-bias polling data subject to SDR.

In an experimental framework, we will test the accuracy of subjects tasked with predicting a series of choices potentially subject to SDR. Our experiment will have two treatments. In Treatment 1, subjects will observe unincentivized, stated-preference data. In Treatment 2, subjects will observe incentivized, revealed-preference data. While Treatment 1 will be subject to both SDR and sampling errors, Treatment 2 will only be subject to sampling errors. Thus, differences in accuracy will be attributable to differences in the magnitude of bias from SDR. With perfect sophistication, subjects in the two treatments should be equally accurate. However, with partial or fully naive subjects, predictions from Treatment 1 will manifest some of the SDR from the underlying data and will be less accurate. We will measure accuracy in terms of the magnitude and direction of the prediction errors.
External Link(s)

Registration Citation

Citation
Brownback, Andy, Nathaniel Burke and Tristan Gagnon-Bartsch. 2023. "Predicting Biased Polls." AEA RCT Registry. October 04. https://doi.org/10.1257/rct.5186-3.1
Sponsors & Partners
Experimental Details

Interventions

Intervention(s)
Subjects in our experiment will be making predictions about a series of choices potentially subject to SDR. Different subjects will receive different information prior to making their predictions.
Intervention (Hidden)
We will begin by recruiting subjects to one of two "choice" groups: "Hypothetical Choice" and "Incentivized Choice." Both groups will respond to a series of eight questions about actions they may wish to take. These actions will be potentially susceptible to SDR: stealing, donating (x2), political candidate donations (x2), political party donations (x2), reducing a donation to a charity for their benefit.

In the Hypothetical Choice group, all respondents will state their preferences for each action. They will have no incentive to state preferences honestly. In the Incentivized Choice group, all respondents will make consequential choices that reveal their preferences for each action. One of their choices will be selected at random to truly count, providing incentives for them to reveal their preferences honestly.

We will then test the accuracy of subjects tasked with predicting the behavior of the Incentivized Choice group. For these predictions, we will retain all subjects from both the Hypothetical and Incentivized Choice groups as well as recruiting fresh participants to compare accuracy from in- and out-of-sample subjects.

Our experiment will have two treatments, each of which will begin by making "Prior" predictions about the decisions of the Incentivized Choice group on each of the eight actions.

After revealing their Prior predictions, all subjects will receive information about responses to each of the eight actions. Subjects will then make "Posterior" predictions. The only difference between the two treatments will be the source of the information.

In Treatment 1, subjects will receive information from a sample of ten respondents from the Hypothetical Choice group.

In Treatment 2, subjects will receive information from a sample of ten respondents from the Incentivized Choice group.

While Treatment 1 will be subject to both SDR and sampling errors, Treatment 2 will only be subject to sampling errors. Thus, differences in accuracy will be attributable to how much of the bias from SDR that manifests itself in the predictions of Treatment 1.

With perfect sophistication, subjects in the two treatments should be equally accurate. However, with partial or fully naive subjects, predictions from Treatment 1 will manifest some of the SDR from the underlying data and will be less accurate. We will measure accuracy in terms of the magnitude and direction of the prediction errors.
Intervention Start Date
2020-02-12
Intervention End Date
2020-06-01

Primary Outcomes

Primary Outcomes (end points)
Our primary outcomes of interest are:

The accuracy of the final predictions: Both the direction and magnitude of the difference between the prediction (Prior or Posterior) and the truth.

The weight placed on each source of information: How much subjects update their predictions based on the information received.

We will look at the differences in each of these outcomes based on:

Receiving information from the Hypothetical or Incentivized Choice groups
Whether the information you received was in the socially-desirable direction
The interaction of information source and social-desirability.
Being previously assigned to the Hypothetical Choice group, Incentivized Choice group, neither group (but from the University of Arkansas), or neither group (recruited from MTurk)
Whether the information you received was from the treatment to which you were assigned
Primary Outcomes (explanation)
Accuracy will be measured in two ways:

The raw difference between the prediction and the truth.
The absolute value of the difference between the prediction and the truth.

Updating will be measured by taking the difference between the Posterior and Prior predictions.

Social desirability of an action will be measured by an out-of-sample survey asking about perceptions towards the action.

Secondary Outcomes

Secondary Outcomes (end points)
We will also evaluate unincentivized predictions about actions that may be subject to SDR, but we will not observe directly (e.g. unsafe sex). These predictions will be part of a calibration exercise where we use the observed ability of subjects to de-bias information from the Hypothetical choice group to improve predictions about actions subject to SDR.
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
Subjects in our experiment will be making predictions about a series of choices potentially subject to SDR. Different subjects will receive different information prior to making their predictions.
Experimental Design Details
We will begin by recruiting subjects to one of two "choice" groups: "Hypothetical Choice" and "Incentivized Choice." Both groups will respond to a series of eight questions about actions they may wish to take. These actions will be potentially susceptible to SDR: stealing, donating (x2), political candidate donations (x2), political party donations (x2), reducing a donation to a charity for their benefit.

In the Hypothetical Choice group, all respondents will state their preferences for each action. They will have no incentive to state preferences honestly. In the Incentivized Choice group, all respondents will make consequential choices that reveal their preferences for each action. One of their choices will be selected at random to truly count, providing incentives for them to reveal their preferences honestly.

We will then test the accuracy of subjects tasked with predicting the behavior of the Incentivized Choice group. For these predictions, we will retain all subjects from both the Hypothetical and Incentivized Choice groups as well as recruiting fresh participants to compare accuracy from in- and out-of-sample subjects.

Our experiment will have two treatments, each of which will begin by making "Prior" predictions about the decisions of the Incentivized Choice group on each of the eight actions.

After revealing their Prior predictions, all subjects will receive information about responses to each of the eight actions. Subjects will then make "Posterior" predictions. The only difference between the two treatments will be the source of the information.

In Treatment 1, subjects will receive information from a sample of ten respondents from the Hypothetical Choice group.

In Treatment 2, subjects will receive information from a sample of ten respondents from the Incentivized Choice group.

While Treatment 1 will be subject to both SDR and sampling errors, Treatment 2 will only be subject to sampling errors. Thus, differences in accuracy will be attributable to how much of the bias from SDR that manifests itself in the predictions of Treatment 1.

With perfect sophistication, subjects in the two treatments should be equally accurate. However, with partial or fully naive subjects, predictions from Treatment 1 will manifest some of the SDR from the underlying data and will be less accurate. We will measure accuracy in terms of the magnitude and direction of the prediction errors.
Randomization Method
To divide subjects into the Hypothetical and Incentivized Choice groups, we will randomly divide our subject pool into two parts and recruit subjects for the groups separately. Recruitment will be identical for the two groups. Subjects will enroll through the SONA recruitment system.

At the prediction stage, Qualtrics will randomly assign subjects to receive information based on the Hypothetical or Incentivized Choice groups. We will block "Predictors" based on their prior experience (Hypothetical Choice group, Incentivized Choice group, or neither) This process will be mechanically random and balanced in sample sizes.
Randomization Unit
Each subject will be randomly assigned to receive information from the Hypothetical or Incentivized choice groups.
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
800 subjects
Sample size: planned number of observations
300 subjects (with 1000 extra in a planned extension)-- 100 Hypothetical Choice subjects from the University of Arkansas 100 Incentivized Choice subjects from the University of Arkansas 100 new subjects from the University of Arkansas who make predictions only (planned extension) 1000 subjects from Amazon's Mechanical Turk who make predictions only
Sample size (or number of clusters) by treatment arms
150 subjects receiving information from the Hypothetical Choice group.
150 subjects receiving information from the Incentivized Choice group.

(Planned extension)
500 subjects receiving information from the Hypothetical Choice group.
500 subjects receiving information from the Incentivized Choice group.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Based on pilot data, each of our tests will have more than twice the necessary sample size to test our primary hypotheses. We chose this number, rather, based on the desire to be conservative, aesthetic preferences for round-numbers, and budget constraints.
Supporting Documents and Materials
IRB

Institutional Review Boards (IRBs)

IRB Name
University of Arkansas Institutional Review Board
IRB Approval Date
2017-11-28
IRB Approval Number
1711081926
Analysis Plan

Analysis Plan Documents

Polling_Predictions_PAP

MD5: a3c8da69dc5aa24621c6b0d46fbe2f5e

SHA1: 6253eafe5980506beb0c66d41e4e3544ebb3646f

Uploaded At: February 11, 2020

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
Yes
Intervention Completion Date
March 31, 2020, 12:00 +00:00
Data Collection Complete
Yes
Data Collection Completion Date
March 31, 2020, 12:00 +00:00
Final Sample Size: Number of Clusters (Unit of Randomization)
39 sentiment group subjects, 282 prediction group subjects
Was attrition correlated with treatment status?
No
Final Sample Size: Total Number of Observations
271 Predictors and 2168 predictions
Final Sample Size (or Number of Clusters) by Treatment Arms
135 Predictors and 1080 predictions with IC data; 136 Predictors and 1088 predictions with H data
Data Publication

Data Publication

Is public data available?
No

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Program Files

Program Files
No
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials