Towards a Cheap Personalization of Political Messaging with Large Language Models?

2024-09-27

2024-10-11

We ask participants for their stance on the policy at hand, for their agreement with the arguments in the message, for the importance they attribute to the topic, how much the message resonates with their political opinion, how much they identify with the message, and how easy or hard the message was to read. Furthermore, we give them the option to generate donations for or against the policy proposed in the message by solving up to ten counting quizzes (the quizzes are voluntary and do not affect the participants' own remuneration).

For each message, we make the LLM explain its "reasoning" variable-by-variable. That is, for every variable (e.g., age), the LLM explains why the message should be appealing to somebody who is young (if the message was tailored to a young person). At the end of the survey, participants choose which aspect (if any) they find most or least convincing. Furthermore, we ask participants at the end of the survey whether they think that an artificial intelligence, a human, or both of them authored the post together.

The reasoning question serves to understand whether the LLM's proposed reasons why the message should be convincing to a person with a certain characteristic (e.g., who is young) are agreed upon by participants who actually do have that characteristic. The question on human vs. AI authorship serves to analyze whether personalization affects the perceived authorship.

We first randomly assign each participant to one out of six policy topic-stances (three topics with a pro/contra statement each). Afterward, each participant is shown one non-personalized, partly personalized, or fully personalized political message on the assign topic-stance. We then evaluate whether (more) personalized messages are more persuasive.

Not available

Randomization within survey (by a computer)

Individual

No

5000 individuals

5000 individuals

Treatment is not binary. See the intervention for an explanation. For example, for a participant who identifies as a Democrat or Republican, the message they are shown will match their partisan affiliation with a 50% probability. Similarly, in terms of age, the probability of the message being matched is 33% (considering that the categories are young, middle-aged, old). Since the message is randomly drawn from all possible messages, all of the covariates being matched is rare: 50% (ideology with Liberal/Conservative) * 50% (political partisanship with Democrat/Republican) * 20% (race with Black, White, Hispanic, Asian, Native American) * 50% (gender with male/female) * 33% (financial status with poor, middle-class, upper-class) * 33% (age with young, middle-aged, old) = 0.28%. For categorical variables, note that, in practice, some participants will likely be part of a category not considered in our prompts (e.g., those whose partisanship is "Independent"). Accordingly, the share of those with a match for the categorical variables is, in expectation, slightly lower than what the calculation above suggests. For the continuous variables, we will follow our implementation partner's mapping into categories and evaluate its robustness to other cutoffs.