Dynamic AI Adoption under Anthropomorphic Beliefs

We will run a survey experiment to test predictions of an AI adoption model. Subjects will see various mathematical tasks, for which we have a measure of human difficulty, and will have to choose whether to delegate each task to a human or to an AI. Tasks can be of two types, with one type harder than the other. They learn the performance of their pick (success or failure) right after. Delegation is incentivized, as subjects gain a small bonus for each success. Subjects repeat this process for around 60 questions, to give them enough signals of human and AI performance. Then, subjects need to make an adoption decision: choose either a human or the AI to solve around 10 random tasks from each type, also incentivized.
We will also elicit incentivized beliefs in performance through the process, as well as a few controls and data quality checks. The survey is designed to take around 20 minutes.
The treatment will manipulate subjects' mental model as it relates to patterns of AI performance.

2024-07-28

2024-08-31

Adoption decisions are the main outcome. There are blue tasks (easier) and green tasks (harder). AI success rate is held constant across the two types of tasks, so that humans dominate AI in the easier tasks and AI dominates humans in the harder ones.
We predict that under the anthropomorphic treatment, the share of "full adoption" (AI chosen for both types of tasks) is higher than under "black box". The share of optimal adoption (adopt AI only for the harder tasks) should be lower in "anthropomorphic" than in the "black box" treatment (we expect this difference to be less marked than for full adoption). Beliefs in performance should reflect this decision, although we may expect a slight taste preference for humans, holding fixed beliefs.

The main sample analysis will drop and replace respondents who fail the simple comprehension question presented in introduction and the attention check presented at the end. Extremely rushed survey answers (among the bottom centiles of time taken) may also be excluded from the main analysis.

We will look into delegation decisions as secondary outcomes. We expect that subjects mostly delegate harder tasks to AI and easier tasks to humans. We expect learning to be more efficient in "black box", as the treatment effectively makes subjects more agnostic regarding the success rate of the AI in both types of tasks. This relatively more efficient learning should be reflected in delegation decisions and beliefs.
For heterogeneity analysis, we will look at treatment effects by degree of prior familiarity with AI. We expect that subjects who are very familiar with AI will see a smaller treatment effect, which we interpret as a lesser effectiveness of the treatment, given that their prior mental model is made more robust with experience. This should also hold true for subjects in the "black box" treatment who suspect the black box to be an AI.

The treatment tries to vary the mental model that subjects are holding regarding AI performance. The treatment consists of changing the framing (without deception) around the AI (which is ChatGPT, a LLM). In the "anthropomorphic" treatment, we present the AI using several human-like traits. In the "black box" treatment, we present the AI as a black box and present it as much as possible as a passive, un-human-like agent. These are the only treatment differences.

Qualtrics survey randomization (block level and javascript) for tasks shown and human performance revealed.

Individual survey respondent

No

Around 300-320 participants after dropping and replacing those failing comprehension question and attention check.

Around 300-320 participants for the final sample used in the main analysis. Around 400 subjects will be initially recruited to drop and replace those failing comprehension/attention check.

Around 150-160 subjects per treatment arm (anthropomorphic and black box).