AI, Information Processing and Dissemination

2022-09-01

2022-09-30

Please answer the following questions:
The three questions refer to the article you just read. You will receive $0.50 for each correctly answered question.

Control Question 1 (Covid): The article states that the virus is still infecting around 150,000 people each day. (FALSE)
Control Question 2 (Covid): The article states that at the peak of the Omicron wave, there were more than 2,600 deaths per day in the U.S. (CORRECT)
Control Question 3 (Covid): The article states that the new subvariant BA.2.75 recently started spreading rapidly in China. (FALSE)

Control Question 1 (Stock market): The article states that the Cboe Volatility Index, or the VIX, recently denoted 25. (FALSE)
Control Question 2 (Stock market): The article states that investors are concerned that inflation will lead to lower consumer spending, which makes up 90% of the U.S. economy. (FALSE)
Control Question 3 (Stock market): The article states that experts expect inflation to stay above 7% for the next few years. (FALSE)


Attitudes toward and perceptions of the article:
1. This article is very helpful to better anticipate the development of the U.S. Total Stock Market Index (Dow Jones) / number of Covid 19 cases in the U.S. over the next month. (1 strongly disagree, 7 strongly agree)
2. Someone who reads this article will be more likely to make an accurate forecast of the development of the U.S. Total Stock Market Index (Dow Jones) / number of Covid 19 cases in the U.S. than someone who does not read it. (1 strongly disagree, 7 strongly agree)
3. The article concentrates on important occurrences, highly relevant to the development of the U.S. Total Stock Market Index (Dow Jones) / number of Covid 19 cases in the U.S., rather than featuring miscellaneous. (1 strongly disagree, 7 strongly agree)
4. The article presents substantial background information highly relevant to the development of the U.S. Total Stock Market Index (Dow Jones) / number of Covid 19 cases. (1 strongly disagree, 7 strongly agree)
5. The article focuses on arguments over emotions in its coverage. (1 strongly disagree, 7 strongly agree)
6. The article offers a comprehensive overview of all the important events. (1 strongly disagree, 7 strongly agree)
7. The article provides objective news. (1 strongly disagree, 7 strongly agree)
8. I generally trust the information presented in the article I just read. (1 strongly disagree, 7 strongly agree)
9. I generally distrust the information presented in the article I just read. (1 strongly disagree, 7 strongly agree)
10. On a scale of -3 to +3, with negative numbers representing left leaning or liberal skew, positive numbers representing right leaning or conservative skew, and 0 representing true neutral, how would you rate the article? (-3 extremely left leaning, 3 extremely right leaning)
11. On a scale of 1 to 7, 1 being not reliable at all and 7 being very reliable, how would you rate the information in the article? (1 not reliable at all, 7 extremely reliable)
12. On a scale of 1 to 7, with 1 being not inspiring at all and 7 being extremely inspiring, how would you rate the article you just read? (1 not inspiring at all, 7 extremely inspiring)
13. If someone is interested in the development of the U.S. Total Stock Market Index (Dow Jones) / number of Covid 19 cases in the U.S. over the next month, how likely is it that you would recommend that article to them? (0% to 100%)

We ask participants to read an article that either contains information about the development of the US stock market index Dow Jones or the number of Covid cases in the US. After participants have read the article, they answer incentivized control questions about the article’s content, trust in the information, and report their perception about the article’s objectivity, quality, leaning, and reliability (we use adapted version of the questions by Bachmann et al. 2022 and self-written items). Additionally, we ask participants to rate whether the information in the article is useful to forecast how the Dow Jones or the number of Covid cases develops over the next month. As their final task, participants have to answer several questions on their demographics, political orientation, and preferences.

The human-generated articles are curated texts from major US news outlets (e.g., CNN, US Today News), where we combine different text pieces. To transform the human-generated information, we will use the Generative Pre-trained Transformer 3 (GPT-3), a state-of-the-art natural language processing (NLP) model. The AI-generated articles are paraphrasings from the original human-generated article that the GPT-3 model produces.

In a between-subject fashion we vary participants’ knowledge about the source of the information. Overall, we employ 6 treatments where we either do not reveal the source (and the set of possible sources), where we correctly reveal the source, or where we incorrectly reveal the opposite source:

T1: human-generated, correct disclosure
T2: AI-generated, correct disclosure
T3: human-generated, incorrect disclosure
T4: AI-generated, incorrect disclosure
T5: human-generated, opacity (no knowledge about possible source)
T6: AI-generated, opacity (no knowledge about possible source)

In our analyses we will compare how the labelling affects participants’ perception about the quality, leaning, and usefulness of the information. At the end of the study, we debrief participants in treatments T3 and T4 and inform them that the article they have read was actually generated by a human / AI.

Treatment texts for AI label:
Source: This text is written by an Artificial Intelligence (AI) System called Generative Pre-trained Transformer 3 (GPT-3). The system is among the most powerful language models that exist today. The quality of the text generated by GPT-3 is so high that it can be difficult to determine whether it was written by a human or an AI.

Treatment texts for human label:
Source: This information are text pieces from articles by professional journalists who regularly write articles for major news outlets.

Discussion of deception (providing incorrect label information):
In treatments 3 and 4 of our first study, we incorrectly inform participants that an AI or a human wrote the article even though it originates from the opposite source. In the broadest sense, this misinformation constitutes deception, although participants’ payoff does not depend on this information in any way. We need this treatment to test for the presence of mere labeling effects. More specifically, it may be possible, that it is not an article's content that creates treatment effects. Instead, differences in perceptions of AI and human-generated articles could merely result from the information that an AI or a human wrote the article. As a result, participants would possess significantly different perceptions of the same article, conditional on its source. To test this conjecture, we need treatments 3 and 4 as additional control treatments against which we can compare results from our other (main) treatment conditions.

Randomization done by survey software.

Individual

No

600 individuals

600 individuals

100 individuals per treatment