Lost in Transmission

The full experiment comprises two separate data collections that build on each other, a transmitter experiment and a listener experiment. The two experimental collections rely on different respondent samples.

TRANSMITTER EXPERIMENT:

Participants listen to two short recordings played consecutively and without a break, each one of an opinion piece providing a qualitative narrative about the future path of a different economic variable. Then, they record their own summary of these recordings, separately for the first and second variable.

LISTENER EXPERIMENT:

This involves a separate set of respondents. For each of the two topics, respondents first state their prior belief about the outcome variable of interest and then listen to a recording about the variable. After listening to a recording, respondents are incentivized to forecast the future development of the variable as well as to guess the prediction of the message originator and the reliability of the original message.

2023-09-14

2023-12-31

Our experiments involve transmission of information about two unknown states: (i) the change in home price growth in a US city and (ii) the change in revenue growth of a US retailer, both for the upcoming year.

Each state has three key dependent variables associated with it: the respondent’s belief movement about the unknown state (difference between posterior, elicited after the respondent listens to a message about the state, and prior, elicited beforehand); the respondent’s belief about the message originator’s prediction about the state of the world; and the respondent’s belief about the originator’s reliability, as measured on a scale ranging from 0 (extremely unreliable) to 100 (extremely reliable).

For the state beliefs, we will drop outliers (as specified below), and the question entry box forces answers to be between -100% and +100%. For all three outcomes, we will z-score within topic*manipulation type quadrants and then pool together the observations from both topics.

All three beliefs are incentivized through random incentives. Respondents will be told at the start that 1 in 10 respondents will be randomly chosen to be eligible for a bonus payment and have one of the incentivized tasks be paid out according to the formulas below.
(a) Beliefs about the true state of the world are incentivized with the following formula:
(i) Probability of winning $20 [in %] = 100 − 10*(Estimate [in %] - True state of the world in 12 months [in %] )^2
(b) Beliefs about the message originator’s beliefs and reliability are incentivized as follows:
(i) For 50% of respondents, there will be no incentive and the question will be phrased as a direct question about the message originator’s beliefs.
(ii) For 50% of respondents, the question will be phrased as a second-order question (“your job is to predict what people would on average respond to the direct question”) and responses will be incentivized with the following formula:
(1) For beliefs about the originator’s prediction: Probability of winning $20 [in %] = 100 − 10*(Response [in %] - Average response to direct question [in %] )^2
(2) For beliefs about the originator’s reliability: Probability of winning $20 [in %] = 100 − 2*(Response - average response to the direct question )^2

If a respondent is selected to be eligible for a bonus, one of the incentivized beliefs of that respondent will be randomly selected to be the one that counts for payment.

The full experiment comprises two separate data collections that build on each other, a transmitter experiment and a listener experiment. The two experimental collections rely on different respondent samples.

TRANSMITTER EXPERIMENT:
Participants listen to two short recordings played consecutively and without a break, each one of an opinion piece providing a qualitative narrative about the future path of a different economic variable. Then, they record their own summary of these recordings, separately for the first and second variable.

A randomly chosen 50% of transmitters will be asked their prior belief about each variable before hearing the recordings, and all transmitters will be asked for the three beliefs described above after recording their transmitted message for each topic.

Recording treatment arms:
Within each topic, we randomize three key features of the original recordings:

*Level of variable*: We randomize whether the piece argues for an increase or a decrease in the level of the variable.

*Reliability of message*: Second, we randomize the reliability of the original message. We randomly assign respondents to one of two different types of reliability manipulations:
- Naturalistic (combination of explicit statements about confidence, source quality and speaker competence, as well as implicit markers of reliability): Respondents in the naturalistic condition are assigned to one of the following 2 conditions: (i) Strong reliability; (ii) Weak reliability.
- Modular (Insertion of explicit markers indicating high or low reliability (e.g., definitely vs. possibly, will vs. might, etc.): Respondents in the modular condition are assigned to one of the following 3 conditions: (i) Strong reliability; (ii) No reliability markers; (iii) Weak reliability.

*Sex of transmitter voice*: We randomize whether the recording is a male voice or a female voice. This is not a focus of analysis and we randomize simply for symmetry.

Randomization is stratified: each transmitter hears two recordings, one with an “increase” and one with a “decrease,” one with “strong reliability” and one with “weak reliability,” and one with a male voice and one with a female voice. Then, if exactly one of the two topics is in the modular condition, that topic has a 33% chance of getting switched to “no reliability markers.” If both topics are in the modular condition, there is a 66% chance that one of the two topics is randomly switched to “no reliability markers.”

Respondents receive incentives for transmitting all information contained in the original messages. Respondents are informed that one in 10 people will be selected for bonus eligibility and that, if selected, a different group of participants will score their recordings on a scale of 0 to 10, where 0 corresponds to “Nothing conveyed in meaning” and 10 corresponds to “Everything conveyed in meaning”. If the average score their recordings receive is at least an 8, they will receive a $20 bonus payment. Between subjects, we randomly assign respondents to two variants of the incentive scheme:

In the “Joint incentive treatment” participants are given the following instructions:
“The other participants will answer the following question about your voice message. How accurately did the voice message convey the content and meaning of what the speaker said?”

In the “Separate incentive treatment”, respondents are explicitly told that the other participants will answer two questions, one about the point forecast implied by the message and one about the reliability of the message. In particular, they receive the following instruction: “The other participants will answer two questions about your voice message.
How accurately was the level of the speaker's prediction conveyed in the voice message?
How accurately was the reliability of the prediction conveyed in the voice message?”

LISTENER EXPERIMENT:

This involves a separate set of respondents. For each of the two topics, respondents first state their prior belief about the outcome variable of interest and then listen to a recording about the variable. As before, the order of the topics is randomized. For each topic, respondents are randomly matched to a transmitter and listen either to the same original recording as the one the transmitter heard, or that transmitter’s transmitted recording. There is a 30% chance of hearing the original and 70% chance of hearing the transmitted recording.

After listening to a recording, respondents are incentivized to forecast the future development of the variable as well as to guess the prediction of the message originator and the reliability of the original message. (The same 3 outcomes described above, incentivized in the same way).

Randomization done by experimental software (Qualtrics)

Individual

No

In the transmitter experiment, we plan to collect data from 500 individuals.
In the listener experiment, we plan to collect data from 1500 individuals.

In the transmitter experiment, we plan to collect data from 500 individuals. In the listener experiment, we plan to collect data from 1500 individuals.

TRANSMITTER EXPERIMENT
- Randomization is stratified: each transmitter hears two recordings, one with an “increase” and one with a “decrease,” one with “strong reliability” and one with “weak reliability,” and one with a male voice and one with a female voice. Then, if exactly one of the two topics is in the modular condition, that topic has a 33% chance of getting switched to “no reliability markers.” If both topics are in the modular condition, there is a 66% chance that one of the two topics is randomly switched to “no reliability markers.”
- Between subjects, we randomly assign transmitters to two variants of the incentive scheme (50% each) as explained above.
- A randomly chosen 50% of transmitters will be asked their prior belief about each variable before hearing the recordings, and all transmitters will be asked for the three beliefs described above after recording their transmitted message for each topic.


LISTENER EXPERIMENT:
- The order in which listeners see the topics is randomized.
- For each topic, respondents are randomly matched to a transmitter and listen either to the same original recording as the one the transmitter heard, or that transmitter’s transmitted recording. There is a 30% chance of hearing the original and 70% chance of hearing the transmitted recording.
- For 50% of respondents, there will be no incentive and the question will be phrased as a direct question about the message originator’s beliefs. For 50% of respondents, the question will be phrased as a second-order question (“your job is to predict what people would on average respond to the direct question”) and responses will be incentivized as described above.