Towards a Cheap Personalization of Political Messaging with Large Language Models?

We expose participants to a LLM-generated political message (either pro or contra) on one of these topics: funding for veteran education and training, introduction of universal healthcare, or changes to the minimum wage. For each topic, we generate a set of messages, where each message is targeted at a demographic subgroup. For the message generation (prompting), subgroups are defined along six variables (with the possible values we consider in parentheses): ideology (Liberal/Conservative), political partisanship (Democrat/Republican), race (Black, White, Hispanic, Asian, Native American), gender (male/female), financial status (poor, middle-class, upper-class), age (young, middle-aged, old).
To understand differences across LLMs, for the healthcare and the minimum wage, half of the messages are generated by ChatGPT and Claude Haiku, each. For the veteran topic, all messages are generated by ChatGPT due to the high refusal rate by Claude Haiku.
When participating in our survey experiment, each participant is first randomly assigned to a topic-stance (e.g., the introduction of universal healthcare-pro or increase of the minimum wage-contra). Then, they are randomly assigned to a message. Hence, by random chance, none, some, or all covariates will match the participant's own background.

2024-09-27

2024-10-11

We ask participants for their stance on the policy at hand, for their agreement with the arguments in the message, for the importance they attribute to the topic, how much the message resonates with their political opinion, how much they identify with the message, and how easy or hard the message was to read. Furthermore, we give them the option to generate donations for or against the policy proposed in the message by solving up to ten counting quizzes (the quizzes are voluntary and do not affect the participants' own remuneration).

For each message, we make the LLM explain its "reasoning" variable-by-variable. That is, for every variable (e.g., age), the LLM explains why the message should be appealing to somebody who is young (if the message was tailored to a young person). At the end of the survey, participants choose which aspect (if any) they find most or least convincing. Furthermore, we ask participants at the end of the survey whether they think that an artificial intelligence, a human, or both of them authored the post together.

The reasoning question serves to understand whether the LLM's proposed reasons why the message should be convincing to a person with a certain characteristic (e.g., who is young) are agreed upon by participants who actually do have that characteristic. The question on human vs. AI authorship serves to analyze whether personalization affects the perceived authorship.

We first randomly assign each participant to one out of six policy topic-stances (three topics with a pro/contra statement each). Afterward, each participant is shown one non-personalized, partly personalized, or fully personalized political message on the assign topic-stance. We then evaluate whether (more) personalized messages are more persuasive.

Randomization within survey (by a computer)

Individual

No

5000 individuals

5000 individuals

Treatment is not binary. See the intervention for an explanation. For example, for a participant who identifies as a Democrat or Republican, the message they are shown will match their partisan affiliation with a 50% probability. Similarly, in terms of age, the probability of the message being matched is 33% (considering that the categories are young, middle-aged, old). Since the message is randomly drawn from all possible messages, all of the covariates being matched is rare: 50% (ideology with Liberal/Conservative) * 50% (political partisanship with Democrat/Republican) * 20% (race with Black, White, Hispanic, Asian, Native American) * 50% (gender with male/female) * 33% (financial status with poor, middle-class, upper-class) * 33% (age with young, middle-aged, old) = 0.28%. For categorical variables, note that, in practice, some participants will likely be part of a category not considered in our prompts (e.g., those whose partisanship is "Independent"). Accordingly, the share of those with a match for the categorical variables is, in expectation, slightly lower than what the calculation above suggests. For the continuous variables, we will follow our implementation partner's mapping into categories and evaluate its robustness to other cutoffs.