Lost in Transmission

Last registered on June 26, 2024

Pre-Trial

Trial Information

General Information

Title
Lost in Transmission
RCT ID
AEARCTR-0012119
Initial registration date
September 14, 2023

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
September 20, 2023, 10:20 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
June 26, 2024, 4:05 PM EDT

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Region

Primary Investigator

Affiliation
Harvard University

Other Primary Investigator(s)

PI Affiliation
University of Cologne
PI Affiliation
Massachusetts Institute of Technology

Additional Trial Information

Status
Completed
Start date
2023-09-14
End date
2023-12-31
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
We examine whether the process of transmitting information through speech distorts the information and how such distortions differ between different kinds of information content. In particular, we hypothesize that distortions are larger for content about the reliability of a message than content about the level of a variable.
External Link(s)

Registration Citation

Citation
Graeber, Thomas, Shakked Noy and Chris Roth. 2024. "Lost in Transmission." AEA RCT Registry. June 26. https://doi.org/10.1257/rct.12119-1.5
Experimental Details

Interventions

Intervention(s)
The full experiment comprises two separate data collections that build on each other, a transmitter experiment and a listener experiment. The two experimental collections rely on different respondent samples.

TRANSMITTER EXPERIMENT:

Participants listen to two short recordings played consecutively and without a break, each one of an opinion piece providing a qualitative narrative about the future path of a different economic variable. Then, they record their own summary of these recordings, separately for the first and second variable.

LISTENER EXPERIMENT:

This involves a separate set of respondents. For each of the two topics, respondents first state their prior belief about the outcome variable of interest and then listen to a recording about the variable. After listening to a recording, respondents are incentivized to forecast the future development of the variable as well as to guess the prediction of the message originator and the reliability of the original message.
Intervention Start Date
2023-09-14
Intervention End Date
2023-12-31

Primary Outcomes

Primary Outcomes (end points)
Our experiments involve transmission of information about two unknown states: (i) the change in home price growth in a US city and (ii) the change in revenue growth of a US retailer, both for the upcoming year.

Each state has three key dependent variables associated with it: the respondent’s belief movement about the unknown state (difference between posterior, elicited after the respondent listens to a message about the state, and prior, elicited beforehand); the respondent’s belief about the message originator’s prediction about the state of the world; and the respondent’s belief about the originator’s reliability, as measured on a scale ranging from 0 (extremely unreliable) to 100 (extremely reliable).

For the state beliefs, we will drop outliers (as specified below), and the question entry box forces answers to be between -100% and +100%. For all three outcomes, we will z-score within topic*manipulation type quadrants and then pool together the observations from both topics.

All three beliefs are incentivized through random incentives. Respondents will be told at the start that 1 in 10 respondents will be randomly chosen to be eligible for a bonus payment and have one of the incentivized tasks be paid out according to the formulas below.
(a) Beliefs about the true state of the world are incentivized with the following formula:
(i) Probability of winning $20 [in %] = 100 − 10*(Estimate [in %] - True state of the world in 12 months [in %] )^2
(b) Beliefs about the message originator’s beliefs and reliability are incentivized as follows:
(i) For 50% of respondents, there will be no incentive and the question will be phrased as a direct question about the message originator’s beliefs.
(ii) For 50% of respondents, the question will be phrased as a second-order question (“your job is to predict what people would on average respond to the direct question”) and responses will be incentivized with the following formula:
(1) For beliefs about the originator’s prediction: Probability of winning $20 [in %] = 100 − 10*(Response [in %] - Average response to direct question [in %] )^2
(2) For beliefs about the originator’s reliability: Probability of winning $20 [in %] = 100 − 2*(Response - average response to the direct question )^2

If a respondent is selected to be eligible for a bonus, one of the incentivized beliefs of that respondent will be randomly selected to be the one that counts for payment.
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
The full experiment comprises two separate data collections that build on each other, a transmitter experiment and a listener experiment. The two experimental collections rely on different respondent samples.

TRANSMITTER EXPERIMENT:
Participants listen to two short recordings played consecutively and without a break, each one of an opinion piece providing a qualitative narrative about the future path of a different economic variable. Then, they record their own summary of these recordings, separately for the first and second variable.

A randomly chosen 50% of transmitters will be asked their prior belief about each variable before hearing the recordings, and all transmitters will be asked for the three beliefs described above after recording their transmitted message for each topic.

Recording treatment arms:
Within each topic, we randomize three key features of the original recordings:

*Level of variable*: We randomize whether the piece argues for an increase or a decrease in the level of the variable.

*Reliability of message*: Second, we randomize the reliability of the original message. We randomly assign respondents to one of two different types of reliability manipulations:
- Naturalistic (combination of explicit statements about confidence, source quality and speaker competence, as well as implicit markers of reliability): Respondents in the naturalistic condition are assigned to one of the following 2 conditions: (i) Strong reliability; (ii) Weak reliability.
- Modular (Insertion of explicit markers indicating high or low reliability (e.g., definitely vs. possibly, will vs. might, etc.): Respondents in the modular condition are assigned to one of the following 3 conditions: (i) Strong reliability; (ii) No reliability markers; (iii) Weak reliability.

*Sex of transmitter voice*: We randomize whether the recording is a male voice or a female voice. This is not a focus of analysis and we randomize simply for symmetry.

Randomization is stratified: each transmitter hears two recordings, one with an “increase” and one with a “decrease,” one with “strong reliability” and one with “weak reliability,” and one with a male voice and one with a female voice. Then, if exactly one of the two topics is in the modular condition, that topic has a 33% chance of getting switched to “no reliability markers.” If both topics are in the modular condition, there is a 66% chance that one of the two topics is randomly switched to “no reliability markers.”

Respondents receive incentives for transmitting all information contained in the original messages. Respondents are informed that one in 10 people will be selected for bonus eligibility and that, if selected, a different group of participants will score their recordings on a scale of 0 to 10, where 0 corresponds to “Nothing conveyed in meaning” and 10 corresponds to “Everything conveyed in meaning”. If the average score their recordings receive is at least an 8, they will receive a $20 bonus payment. Between subjects, we randomly assign respondents to two variants of the incentive scheme:

In the “Joint incentive treatment” participants are given the following instructions:
“The other participants will answer the following question about your voice message. How accurately did the voice message convey the content and meaning of what the speaker said?”

In the “Separate incentive treatment”, respondents are explicitly told that the other participants will answer two questions, one about the point forecast implied by the message and one about the reliability of the message. In particular, they receive the following instruction: “The other participants will answer two questions about your voice message.
How accurately was the level of the speaker's prediction conveyed in the voice message?
How accurately was the reliability of the prediction conveyed in the voice message?”

LISTENER EXPERIMENT:

This involves a separate set of respondents. For each of the two topics, respondents first state their prior belief about the outcome variable of interest and then listen to a recording about the variable. As before, the order of the topics is randomized. For each topic, respondents are randomly matched to a transmitter and listen either to the same original recording as the one the transmitter heard, or that transmitter’s transmitted recording. There is a 30% chance of hearing the original and 70% chance of hearing the transmitted recording.

After listening to a recording, respondents are incentivized to forecast the future development of the variable as well as to guess the prediction of the message originator and the reliability of the original message. (The same 3 outcomes described above, incentivized in the same way).

Experimental Design Details
Randomization Method
Randomization done by experimental software (Qualtrics)
Randomization Unit
Individual
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
In the transmitter experiment, we plan to collect data from 500 individuals.
In the listener experiment, we plan to collect data from 1500 individuals.
Sample size: planned number of observations
In the transmitter experiment, we plan to collect data from 500 individuals. In the listener experiment, we plan to collect data from 1500 individuals.
Sample size (or number of clusters) by treatment arms
TRANSMITTER EXPERIMENT
- Randomization is stratified: each transmitter hears two recordings, one with an “increase” and one with a “decrease,” one with “strong reliability” and one with “weak reliability,” and one with a male voice and one with a female voice. Then, if exactly one of the two topics is in the modular condition, that topic has a 33% chance of getting switched to “no reliability markers.” If both topics are in the modular condition, there is a 66% chance that one of the two topics is randomly switched to “no reliability markers.”
- Between subjects, we randomly assign transmitters to two variants of the incentive scheme (50% each) as explained above.
- A randomly chosen 50% of transmitters will be asked their prior belief about each variable before hearing the recordings, and all transmitters will be asked for the three beliefs described above after recording their transmitted message for each topic.


LISTENER EXPERIMENT:
- The order in which listeners see the topics is randomized.
- For each topic, respondents are randomly matched to a transmitter and listen either to the same original recording as the one the transmitter heard, or that transmitter’s transmitted recording. There is a 30% chance of hearing the original and 70% chance of hearing the transmitted recording.
- For 50% of respondents, there will be no incentive and the question will be phrased as a direct question about the message originator’s beliefs. For 50% of respondents, the question will be phrased as a second-order question (“your job is to predict what people would on average respond to the direct question”) and responses will be incentivized as described above.


Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Supporting Documents and Materials

Documents

Document Name
Additional information
Document Type
other
Document Description
File
Additional information

MD5: 0ffb659875add7703316bb816082e744

SHA1: 30c86b7979f43a450849594b9d5bc1ae0d95c23e

Uploaded At: September 14, 2023

Document Name
Additional information on study with numerical statements
Document Type
other
Document Description
File
Additional information on study with numerical statements

MD5: 860e3cb1f8be63438b38413d8be36e56

SHA1: 039ec143996579cbefccc0ce932c852651461589

Uploaded At: June 26, 2024

Document Name
Additional information on Study with Incentives Based on Belief Movement
Document Type
other
Document Description
File
Additional information on Study with Incentives Based on Belief Movement

MD5: bd8f81bd04808952a566237d4e07c67d

SHA1: 3213defcf31e696854b0924de431e077aa7396e9

Uploaded At: November 14, 2023

Document Name
Additional information on study about choice of incentives
Document Type
other
Document Description
File
Additional information on study about choice of incentives

MD5: f02aa5c7c3b102f69fb2ff0a87e46825

SHA1: 9648ee6ee5c530be197ba10a57f253f525f3f95e

Uploaded At: November 20, 2023

Document Name
Additional information on study about salience of incentives
Document Type
other
Document Description
File
Additional information on study about salience of incentives

MD5: 70990bab280401fbd3d23ad38bab442a

SHA1: ab03275870f2e3a99eb4f20d73d1b7918b308517

Uploaded At: November 27, 2023

IRB

Institutional Review Boards (IRBs)

IRB Name
Harvard University-Area Committee on the Use of Human Subjects
IRB Approval Date
2023-03-27
IRB Approval Number
IRB23-0080

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials