Model uncertainty and overprecision (TAO-RRT)

Last registered on November 15, 2024
View Trial History

Pre-Trial

Trial Information

General Information

Title
Model uncertainty and overprecision (TAO-RRT)
RCT ID
AEARCTR-0014396
Initial registration date
October 31, 2024

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
November 15, 2024, 1:05 PM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Country
United States of America
Region

Primary Investigator

Name
Don Moore
Affiliation
UC Berkeley
Contact Primary Investigator

Other Primary Investigator(s)

PI Name
Matthew Backus
PI Affiliation
UC Berkeley
Contact Investigator
PI Name
Ned Augenblick
PI Affiliation
UC Berkeley
Contact Investigator
PI Name
Andrew Little
PI Affiliation
UC Berkeley
Contact Investigator

Additional Trial Information

Status
Completed
Start date
2024-10-30
End date
2024-11-15
Keywords
Behavior
Additional Keywords
overconfidence, overprecision, model uncertainty
JEL code(s)
D83, D90, C91
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
Forming beliefs about the world (whether using formal statistical procedures or more generally) usually requires simplifying assumptions. Unfortunately, it is often cognitively costly or even impossible to consider how all possible assumptions would affect the relevant beliefs. We develop a formal model of individuals who properly recognize uncertainty conditional on the assumptions they make (``within-model uncertainty''), but do not recognize how their assumptions affect their beliefs (``across-model uncertainty''). The model's main result is that this leads to overprecision, in the sense of having too low a subjective variance relative to the truth and to the true mean square error of predictions. Making assumptions can also generate disagreement among those with the same information. Further, if assumptions are drawn independently from the correct distribution the amount of disagreement is exactly equal to the amount of overprecision. This study explores these predictions in an experiment that varies within-model and across-model uncertainty.
External Link(s)
This OSF site includes survey and stimulus materials.

Registration Citation

Citation
Augenblick, Ned et al. 2024. "Model uncertainty and overprecision (TAO-RRT)." AEA RCT Registry. November 15. https://doi.org/10.1257/rct.14396-1.0
Sponsors & Partners
Experimental Details

Interventions

Intervention(s)
The experimental task presents participants with a problem in which they make a prediction about the mean and variance of an outcome.
In each of 15 rounds, each participant sees a scatterplot image representing sales data over time. The task is to make a prediction about future sales, using two slider bars: one for the mean and the second for the uncertainty around that mean.
Intervention (Hidden)
Intervention Start Date
2024-10-30
Intervention End Date
2024-11-15

Primary Outcomes

Primary Outcomes (end points)
Our main dependent variable for the analysis is the average reported variance for a particular image. From it we can compute overprecision, the true variance for a condition minus the average reported variance (averaged in the slider bar and then converted to a variance).
Primary Outcomes (explanation)
Our planned analyses will test the prediction that reported variance better reflects within-model than between-model uncertainty.

Secondary Outcomes

Secondary Outcomes (end points)
To compute disagreement for a given scatterplot, we estimate the variance in the mean estimates for that estimate.
Secondary Outcomes (explanation)
We will test the degree to which disagreement between individuals captures between-model uncertainty by regressing the error in the variance estimate on disagreement between individuals’ best-guess forecasts; our theory predicts a regression weight of 1, in contrast to the normative baseline of 0.

Experimental Design

Experimental Design
The 15 scatterplots are drawn from a set of 160 that vary with respect to: 1) the trendline, 2) noisiness of realizations, 3) whether the true trendline is shown on the image, and 4) the temporal distance to the prediction point from the sample data.

The images participants see are constructed from a set of 40 "problems" each of which has four conditions. A problem is a pattern of dots, and a baseline and alternative distance for the future prediction (which may be shorter or longer). The condition then selects one of the distances, and whether the trendline is given. We allow for 4 possible conditions: 1) a baseline problem (with no line), 2) the baseline with the alternative distance to the prediction point, 3) the baseline with the line depicted, 4) the baseline with both the line depicted and noise scaled such that the correct variance of the prediction is equal to that under condition 1.
Experimental Design Details
There are five conditions: 1) a baseline problem (with no line), 2) the baseline with greater or smaller distance to the prediction point, 3) the baseline with the line depicted, 4) the baseline with greater or smaller noise in the variance, and 5) the baseline with both the line depicted and greater or smaller noise in the variance.
The code that generates the 150 images appears on the project's website: https://osf.io/q86ev/
Randomization Method
For the main problems, we first randomize which 12 of the 40 problems they will see, and then randomly select one of the 4 conditions for each problem. We also randomize the order in which these images are shown.
Finally, for secondary analysis we select 3 of the 12 images to be shown again. One of these is an image with a line, one with no line and a relatively low distance, and one is with no line and relatively high distance. These “repeats” are randomly interspersed with the other images.
Randomization Unit
Randomization occurs at the level of the image.
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
We will collect 600 participants from Prolific and then exclude those who run afoul of the exclusion criteria. (This will leave us with less than 600.)
Sample size: planned number of observations
600 participants rating 15 images yields 9000 images rated (before exclusions)
Sample size (or number of clusters) by treatment arms
9000 images clustered within 600 participants and 4 conditions.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
In our October 2023 pilot test (code-named PES), the difference between the regression coefficients for within-model and between-model variance differed by 16 standard errors. Our planned sample yields more than 90% power to detect this effect.
Supporting Documents and Materials

Documents

Document Name
Qualtrics survey
Document Type
survey_instrument
Document Description
File
Qualtrics survey

MD5: e2e917b2d69f316fb5766916943ea53f

SHA1: dd85101badda166e27aeca72e56769f0d4c11177

Uploaded At: October 31, 2024

IRB

Institutional Review Boards (IRBs)

IRB Name
Committee for the Protection of Human Subjects
IRB Approval Date
2012-03-12
IRB Approval Number
2010-11-2549
Analysis Plan

Analysis Plan Documents

RRT pre-analysis plan

MD5: 2865dc7c24540dcfa621a0cb397f06de

SHA1: 6d86aad082d46a91da48af856efdf370dfb63a9b

Uploaded At: October 31, 2024

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials