x

NEW UPDATE: Completed trials may now upload and register supplementary documents (e.g. null results reports, populated pre-analysis plans, or post-trial results reports) in the Post Trial section under Reports, Papers, & Other Materials.
Predicting Biased Polls
Last registered on February 12, 2020

Pre-Trial

Trial Information
General Information
Title
Predicting Biased Polls
RCT ID
AEARCTR-0005186
Initial registration date
February 11, 2020
Last updated
February 12, 2020 1:40 PM EST
Location(s)
Region
Primary Investigator
Affiliation
University of Arkansas
Other Primary Investigator(s)
PI Affiliation
University of Arkansas
PI Affiliation
Harvard University
Additional Trial Information
Status
In development
Start date
2020-02-11
End date
2020-06-01
Secondary IDs
Abstract
Unincentivized surveys of stated-preference may be susceptible to "socially desirable responding" (SDR, hereafter) where respondents strategically conceal preferences that they perceive to be socially undesirable. In contrast, economists often assume that the incentive-compatibility of revealed-preference elicitations mitigates the threat of SDR by imposing real costs. One domain in which SDR appears particularly powerful is the realm of politics where SDR is both ubiquitous and well-known by the popular press and other "consumers" of polling research. Moreover, incentive-compatible elicitations of political preferences are not feasible. However, if awareness of SDR is sufficient to de-bias polling information, then SDR may not cause any negative consequences. This project seeks to test the ability of economic agents to de-bias polling data subject to SDR.

In an experimental framework, we will test the accuracy of subjects tasked with predicting a series of choices potentially subject to SDR. Our experiment will have two treatments. In Treatment 1, subjects will observe unincentivized, stated-preference data. In Treatment 2, subjects will observe incentivized, revealed-preference data. While Treatment 1 will be subject to both SDR and sampling errors, Treatment 2 will only be subject to sampling errors. Thus, differences in accuracy will be attributable to differences in the magnitude of bias from SDR. With perfect sophistication, subjects in the two treatments should be equally accurate. However, with partial or fully naive subjects, predictions from Treatment 1 will manifest some of the SDR from the underlying data and will be less accurate. We will measure accuracy in terms of the magnitude and direction of the prediction errors.
External Link(s)
Registration Citation
Citation
Brownback, Andy, Nathaniel Burke and Tristan Gagnon-Bartsch. 2020. "Predicting Biased Polls." AEA RCT Registry. February 12. https://doi.org/10.1257/rct.5186-1.0.
Experimental Details
Interventions
Intervention(s)
Subjects in our experiment will be making predictions about a series of choices potentially subject to SDR. Different subjects will receive different information prior to making their predictions.
Intervention Start Date
2020-02-12
Intervention End Date
2020-06-01
Primary Outcomes
Primary Outcomes (end points)
Our primary outcomes of interest are:

The accuracy of the final predictions: Both the direction and magnitude of the difference between the prediction (Prior or Posterior) and the truth.

The weight placed on each source of information: How much subjects update their predictions based on the information received.

We will look at the differences in each of these outcomes based on:

Receiving information from the Hypothetical or Incentivized Choice groups
Whether the information you received was in the socially-desirable direction
The interaction of information source and social-desirability.
Being previously assigned to the Hypothetical Choice group, Incentivized Choice group, neither group (but from the University of Arkansas), or neither group (recruited from MTurk)
Whether the information you received was from the treatment to which you were assigned
Primary Outcomes (explanation)
Accuracy will be measured in two ways:

The raw difference between the prediction and the truth.
The absolute value of the difference between the prediction and the truth.

Updating will be measured by taking the difference between the Posterior and Prior predictions.

Social desirability of an action will be measured by an out-of-sample survey asking about perceptions towards the action.
Secondary Outcomes
Secondary Outcomes (end points)
We will also evaluate unincentivized predictions about actions that may be subject to SDR, but we will not observe directly (e.g. unsafe sex). These predictions will be part of a calibration exercise where we use the observed ability of subjects to de-bias information from the Hypothetical choice group to improve predictions about actions subject to SDR.
Secondary Outcomes (explanation)
Experimental Design
Experimental Design
Subjects in our experiment will be making predictions about a series of choices potentially subject to SDR. Different subjects will receive different information prior to making their predictions.
Experimental Design Details
We will begin by recruiting subjects to one of two "choice" groups: "Hypothetical Choice" and "Incentivized Choice." Both groups will respond to a series of eight questions about actions they may wish to take. These actions will be potentially susceptible to SDR: stealing, donating (x2), political candidate donations (x2), political party donations (x2), reducing a donation to a charity for their benefit. In the Hypothetical Choice group, all respondents will state their preferences for each action. They will have no incentive to state preferences honestly. In the Incentivized Choice group, all respondents will make consequential choices that reveal their preferences for each action. One of their choices will be selected at random to truly count, providing incentives for them to reveal their preferences honestly. We will then test the accuracy of subjects tasked with predicting the behavior of the Incentivized Choice group. For these predictions, we will retain all subjects from both the Hypothetical and Incentivized Choice groups as well as recruiting fresh participants to compare accuracy from in- and out-of-sample subjects. Our experiment will have two treatments, each of which will begin by making "Prior" predictions about the decisions of the Incentivized Choice group on each of the eight actions. After revealing their Prior predictions, all subjects will receive information about responses to each of the eight actions. Subjects will then make "Posterior" predictions. The only difference between the two treatments will be the source of the information. In Treatment 1, subjects will receive information from a sample of ten respondents from the Hypothetical Choice group. In Treatment 2, subjects will receive information from a sample of ten respondents from the Incentivized Choice group. While Treatment 1 will be subject to both SDR and sampling errors, Treatment 2 will only be subject to sampling errors. Thus, differences in accuracy will be attributable to how much of the bias from SDR that manifests itself in the predictions of Treatment 1. With perfect sophistication, subjects in the two treatments should be equally accurate. However, with partial or fully naive subjects, predictions from Treatment 1 will manifest some of the SDR from the underlying data and will be less accurate. We will measure accuracy in terms of the magnitude and direction of the prediction errors.
Randomization Method
To divide subjects into the Hypothetical and Incentivized Choice groups, we will randomly divide our subject pool into two parts and recruit subjects for the groups separately. Recruitment will be identical for the two groups. Subjects will enroll through the SONA recruitment system.

At the prediction stage, Qualtrics will randomly assign subjects to receive information based on the Hypothetical or Incentivized Choice groups. We will block "Predictors" based on their prior experience (Hypothetical Choice group, Incentivized Choice group, or neither) This process will be mechanically random and balanced in sample sizes.
Randomization Unit
Each subject will be randomly assigned to receive information from the Hypothetical or Incentivized choice groups.
Was the treatment clustered?
No
Experiment Characteristics
Sample size: planned number of clusters
800 subjects
Sample size: planned number of observations
300 subjects (with 1000 extra in a planned extension)-- 100 Hypothetical Choice subjects from the University of Arkansas 100 Incentivized Choice subjects from the University of Arkansas 100 new subjects from the University of Arkansas who make predictions only (planned extension) 1000 subjects from Amazon's Mechanical Turk who make predictions only
Sample size (or number of clusters) by treatment arms
150 subjects receiving information from the Hypothetical Choice group.
150 subjects receiving information from the Incentivized Choice group.

(Planned extension)
500 subjects receiving information from the Hypothetical Choice group.
500 subjects receiving information from the Incentivized Choice group.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Based on pilot data, each of our tests will have more than twice the necessary sample size to test our primary hypotheses. We chose this number, rather, based on the desire to be conservative, aesthetic preferences for round-numbers, and budget constraints.
IRB
INSTITUTIONAL REVIEW BOARDS (IRBs)
IRB Name
University of Arkansas Institutional Review Board
IRB Approval Date
2017-11-28
IRB Approval Number
1711081926
Analysis Plan

There are documents in this trial unavailable to the public. Use the button below to request access to this information.

Request Information
Post-Trial
Post Trial Information
Study Withdrawal
Intervention
Is the intervention completed?
No
Is data collection complete?
Data Publication
Data Publication
Is public data available?
No
Program Files
Program Files
Reports, Papers & Other Materials
Relevant Paper(s)
REPORTS & OTHER MATERIALS