Policymaker Preferences for Scientific Evidence Synthesis: A Discrete Choice Experiment

This study implements a discrete choice experiment (DCE) as the primary intervention to elicit policymaker preferences for scientific evidence synthesis characteristics. The intervention consists of structured decision-making scenarios designed to reveal underlying preferences through observed choices. The DCE presents Italian policymakers with several binary choice tasks where they select between alternative scientific evidence syntheses to support policy development. Each synthesis option varies across three attributes: delivery time, cost, and production method.
Participants choose within a standardized scenario where they must develop a policy proposal within limited time and need to commission a scientific synthesis. The experimental design randomizes attribute combinations across choice tasks to enable estimation of preferences and willingness-to-pay for different synthesis characteristics.
The discrete choice experiment is complemented with a comprehensive survey that collects participant background information (role, institutional level, geographic location, policy area expertise, activities related to public intervention, demographic characteristics) as well as current evidence use practices and technology attitudes (attitudes toward science-policy collaboration, and AI usage frequency and acceptance). The survey data allows for heterogeneity analysis and examination of relationships between stated and revealed preferences.

This trial is part of a broader study on policymakers’ use of evidence. A second discrete choice experiment, registered separately under the title “Policymaker Preferences for Study Characteristics in Evidence-Based Policy: A Discrete Choice Experiment“ [add trial number or link], explores preferences for different characteristics of scientific studies.

2025-07-22

2025-10-31

- Choice probability by synthesis production method: the probability of selecting expert-generated, AI-generated, or AI-generated with expert supervision syntheses across all choice tasks;
- AI type effect on AI preference: the difference in choice probability for AI-generated syntheses between the validated AI treatment group and the standard AI control group;
- Willingness-to-pay estimates: implicit monetary valuations for different synthesis production methods;
- Marginal rates of substitution: trade-off rates between attributes (e.g., how much additional cost participants accept for faster delivery time, or how much time delay they accept for expert vs AI production);
- attribute importance weights: the relative importance participants place on delivery time, cost, and production method in their synthesis selection decisions.

- Heterogeneity in preferences by policymaker characteristics (political vs. administrative roles, government levels, policy areas, geographical area, demographic characteristics, atititude towards AI and attitudes towards science);
- Correlations between choices and other survey measures (AI usage and attitudes, evidence use patterns, attitudes toward science-policy collaboration);
- Preference class identification: distinct preference segments identified, including characterization of segment membership based on participant characteristics.

Design. This study employs a discrete choice experiment (DCE) with a between and within experimental design.

All participants are exposed to the same vignette (hypothetical scenario): they need to imagine being responsible for developing a policy proposal within a three-month deadline addressing a highly relevant issue in their area of competence. The have a fixed budget for this task, a part of which they can allocate to the production of synthesis of the available scientific evidence for the policy problem.

Choice task structure. Participants complete 8 binary choice tasks, each presenting two alternative scientific evidence syntheses. The two synthesis options are presented side-by-side on desktop computers or vertically stacked on mobile devices, with attributes displayed using visual icons and standardized descriptions. In each task, participants must select their preferred option to support policy development by clicking the corresponding button. Choice tasks are presented one at a time and are self-paced, but must be completed in a single session. Participants cannot return to previous tasks once submitted.

Attribute specification. Each synthesis option is characterized by three attributes with the following levels:

1) Production method:
• Expert-generated: Experts (academics or consultants with experience on the topic) produce a document that synthesizes the scientific evidence
• AI-generated: A generative artificial intelligence (AI) model automatically produces a document that synthesizes the scientific evidence
• AI-generated with expert supervision: A generative artificial intelligence (AI) model automatically produces a document that synthesizes the scientific evidence, whose content is reviewed by experts

2) Delivery time:
• For the Expert-generated production method: 2 weeks, 1 month, 2 months;
• For the AI-generated production method: 1 day, 1 week, 2 weeks;
• For the AI-generated with expert supervision production method: 1 week, 2 weeks, 1 month.

3) Cost:
• For the Expert-generated production method: 1000 € , 1500 €, 2000 €, 3000 €, 4000 €;
• For the AI-generated production method: 250 €, 500 €, 1000 €, 1500 €, 2000 €;
• For the AI-generated with expert supervision production method: 500 €, 1000 €, 1500 €, 2000 €, 3000 €.

Treatment manipulation. Participants are randomly assigned to one of two between-subject conditions that differ in the type of AI used in AI-generated options:
- Treatment group (validated generative AI): the AI model is specifically validated for scientific study synthesis. Participants are informed that "validated" means the AI has been tested by comparing its outputs with those generated by human experts to verify accuracy and reliability.
- Control group (standard generative AI): the AI is a generic generative AI model. Participants are provided with a brief definition of standard generative AI.
Both groups receive identical descriptions for expert-generated syntheses.

The choice set are generated through a constrained randomized design. In each choice set, the main attribute (production method) varies between the two choices. The values of the other two attributes are allowed to vary randomly in each choice set, under the constraints reported above, chosen to ensure realism of the choice.

The data collection will be stagered, we will invite participants gradually, spanning the invitations across several weeks. If the response rate for the first 5000 invitations sent is lower than 3%, we will use the collected data to generate priors and change the design to a D-efficient fractional design, to ensure a higher efficiency for the estimation of the main effects. The intial pool of respondents will be assigned to a separate block and we will constrol for it in the analysis.

Randomization is implemented through oTree's built-in randomization functions (python random module).

The randomization unit is the individual participant. Each participant is independently randomly assigned to one of the two treatment conditions (validated AI vs standard AI) upon entering the oTree session.

No

Each individual policymaker is a unit of analysis. The target is 1,000-1,300 respondents, which corresponds to an 8-10% response rate from the initial 13,000 contact list.

1,000-1,300 respondents, who are asked to perform 8 choice-tasks each.

Equal randomization to treatment/control groups: 500-650 policymakers in treatment group, 500-650 policymakers in control group