AI-Induced Confirmation Bias

For experimental details, see the Experimental Design (Public) section.

2026-08-27

2026-09-30

H1. We hypothesize that the gap between the LLM’s predicted probabilities for participants with “increase” versus “decrease” priors is larger under Memory ON than under Memory OFF.

H2. We hypothesize that the gap between posterior predictions of participants with "increase" versus "decrease" priors is larger under Memory ON than under Memory OFF.

H1. We classify participants by the direction of their prior belief about approval (increase versus decrease). We regress the AI predicted probability on an indicator for "increase" prior, an indicator for Memory ON, and their interaction, controlling for topic fixed effects. Standard errors are clustered at the individual level.

H2. We estimate the same specification using participants' posterior predictions as the dependent variable.

Sample restrictions. All analyses of AI predictions and of participants' posterior predictions omit participant-topic observations in which the participant did not use the AI under an AI treatment, and participant-topic observations for which the participant's prior is not elicited as increase or decrease.

H3. We repeat the primary analyses using political party instead of prior direction as the grouping variable. We test whether the gap between Republican and Democrat predictions is larger under Memory ON than under Memory OFF, for both the AI's predictions and participants' posteriors.

H4. We test whether participants place different weights on their prior prediction and the AI signal under Memory ON versus Memory OFF.

H5. We estimate participants' mean willingness to pay (WTP) for the memory feature. We may explore the correlation of WTP with AI usage in the experiment, treatment effects in H2, and belief updating in H4.

We also consider several exploratory analyses:

E1. We test whether the AI’s predicted probability is higher under Memory ON than under Memory OFF for participants with “increase” priors, and lower under Memory ON than under Memory OFF for participants with “decrease” priors. We test the same for participants’ posterior predictions.

E2. We explore estimating a belief-updating model in log odds to test whether participants act as if they neglect the correlation between AI memory and their prior beliefs.

E3. We explore whether the effects in H2 differ by participants' self-reported AI usage frequency, education level, memory feature behavior, and willingness to pay for the memory feature.

E4. We explore whether participants' AI usage correlates with self-reported AI usage frequency, education level, and memory feature behavior.

E5. We compare the accuracy of predictions across Memory ON, Memory OFF, and no-AI conditions.

H3. We estimate the same regression as in H1 and H2, but replacing the “increase” prior indicator with an indicator for Republican party identification.

H4. We define participants' prior predictions as their counterfactual predictions in the absence of the AI signal. We predict prior predictions using a model trained on no-AI participants, whose posterior predictions serve as prior predictions, and regress posterior predictions on the predicted prior prediction, the AI prediction, a Memory ON indicator, and the interactions of Memory ON with both the predicted prior prediction and the AI prediction, controlling for topic fixed effects. Standard errors are clustered at the individual level.

H5. We report mean WTP from the multiple price list. We define AI usage as whether the participant clicks on the AI prediction for a given topic. We regress AI usage on a Memory ON indicator, WTP, and their interaction, controlling for topic fixed effects. We run similar regressions when interacting with H2 and H4. Standard errors are clustered at the individual level.

Sample restrictions. All analyses of AI predictions and of participants' posterior predictions omit participant-topic observations in which the participant did not use the AI under an AI treatment, and participant-topic observations for which the participant's prior is not elicited as increase or decrease.

This experiment is conducted in two parts. In the first part, participants have a 5-minute conversation with an AI interviewer (GPT-5.6-Sol with default parameters). In the second part, participants make probabilistic predictions about future events. The structure of the experiment is as follows: First, participants are interviewed by the AI about their views on Trump's policies on two topics: trade and the economy. The order of the two topics is randomized within subjects. The interview covers three phases per topic: what the participant finds important, how they feel about it, and where they think public approval is headed. The third phase elicits participants' priors about the direction of approval. Then, participants predict the probability that Trump's approval ratings on trade and on the economy will increase over the next month (i.e., participant posteriors). We describe the prediction process in detail below.

Our main treatment arm, which varies between subjects, is whether participants receive AI assistance with their predictions, and if so, whether the AI has access to their interview conversation:
- Memory ON: the AI prediction assistant has access to the participant's interview conversation.
- Memory OFF: the AI prediction assistant does not have access to the interview conversation.
- No AI: participants make predictions without AI assistance.

Before learning their assignment, participants complete a multiple price list to elicit their willingness to pay for the memory feature. The multiple price list has 11 rows, each offering a choice between Memory OFF (with varying bonus amounts) and Memory ON (with varying bonus amounts). We then randomize whether participants see an AI prediction (80% chance) or are in the No AI condition (20% chance). Then, among the participants assigned to the AI treatments, with 2% probability one row of the multiple price list is randomly selected, and the participant’s choice is implemented; with 98% probability, the memory setting is randomly assigned to be either Memory OFF or Memory ON (with equal chance). We restrict our analyses comparing Memory ON and Memory OFF to participants whose memory assignment is determined by random assignment. Overall, this comes out to 20% facing No AI, 39.2% in Memory OFF, and 39.2% in Memory ON.

For each prediction task, participants see the current approval rating from a polling average and predict the probability (0-100%) that it will increase over the next month. Participants in the AI conditions can click a button to receive AI assistance (GPT-5.6-Sol with default parameters) via a 3-minute chatbot. The AI's first response includes a web search; this first request is by default a request for the AI's prediction, and the AI's response to this request is the AI prediction used in our analyses. For analyses of the AI’s predictions, we complement each prediction delivered in the experiment with a counterfactual prediction generated offline under the opposite memory condition using an identical request, and we may repeat both under additional LLM models. After interacting with the AI (or not), participants enter their prediction and report their confidence. Participants in the AI conditions also report whether they used the AI assistance.

One of the two prediction tasks is randomly selected to determine the participant's bonus payment of up to $2. We use a binarized quadratic scoring rule: if the event occurs, the participant receives the bonus with probability 1 - (1-p)^2, where p is the reported belief; if the event does not occur, the participant receives the bonus with probability 1 - p^2.

Finally, we will ask participants a set of questions, including about demographics, AI usage frequency, and memory feature behavior, which we may use in exploratory analyses.

We will also give participants two attention-check questions and drop them from our analyses if they do not answer them correctly.

Not available

Randomization for all treatments done by computer.

We randomize whether participants receive AI assistance (No AI versus AI) between subjects using blocked randomization with blocks of 5 (1 No AI, 4 AI). We randomize whether AI participants' memory setting is determined by random assignment or by their own price list choice between subjects using blocked randomization with blocks of 100 (98 random assignment, 2 price list). We randomize the memory setting (Memory ON versus Memory OFF) between subjects using blocked randomization with blocks of 2 among participants whose memory assignment is determined by random assignment. We randomize the order of the two prediction topics (trade and economy) within subjects.

We randomize the treatment at the individual level.

No

1000 participants

2000 posterior predictions, 2000 prior directions (1000 participants × 2 topics). At most 4,000 AI predictions per model (at most 2,000 transcripts × 2 memory conditions, one experimental and one counterfactual).

Of the 1,000 participants, we estimate 200 assigned to No AI and 800 to AI assistance. Among the 800 AI participants, 16 (2%) have their memory setting implemented from their own price-list choice; the remaining 784 are randomly assigned, with 392 in Memory OFF and 392 in Memory ON.