Policymaker Preferences for Study Characteristics in Evidence-Based Policy: A Discrete Choice Experiment

This study implements a discrete choice experiment (DCE) as the primary intervention to elicit policymaker preferences for scientific study characteristics in evidence synthesis. The intervention consists of structured decision-making scenarios designed to reveal underlying preferences through observed choices. The DCE presents Italian policymakers with several binary choice tasks where they select between alternative types of scientific studies to be included in evidence syntheses that support policy development. Each study option varies across four attributes: geographic coverage, time period analyzed, collection of direct experiences through interviews, and type of impact evaluation methodology.
Participants choose within a standardized scenario where they must develop a policy proposal within limited time and need to select which types of studies should be prioritized for inclusion in their evidence synthesis. The experimental design randomizes attribute combinations across choice tasks to enable estimation of preferences and relative importance of different study characteristics. Additionally, the study includes a treatment manipulation regarding the institutional decision-making context: participants are randomly assigned to scenarios where commission members are either open to alternative policy approaches (control) or strongly committed to different policy solutions with little openness to alternatives (treatment).
The discrete choice experiment is complemented with a comprehensive survey that collects participant background information (role, institutional level, geographic location, policy area expertise, activities related to public intervention, demographic characteristics) as well as current evidence use practices and technology attitudes (attitudes toward science-policy collaboration, and AI usage frequency and acceptance). The survey data allows for heterogeneity analysis and examination of relationships between stated and revealed preferences.

This trial is part of a broader study on policymakers’ use of evidence. A second discrete choice experiment, registered separately under the title “Policymaker Preferences for Scientific Evidence Synthesis: A Discrete Choice Experiment“ [add trial number or link], explores preferences for different types of scientific evidence syntheses.

2025-07-22

2025-09-15

- Choice probability by study characteristics: the probability of selecting studies with different levels of geographic coverage, time periods, direct experience collection, and impact evaluation methodologies across all choice tasks;
- Treatment effect of institutional climate: differences in preference parameters between collaborative and conflicting institutional climate;
- Marginal rates of substitution: trade-off rates between different study characteristics, measuring how much policymakers are willing to accept changes in one attribute to obtain improvements in another;
- Relative importance weights: the relative importance of each attribute in policymakers' decision-making.

- Heterogeneity in preferences by policymaker characteristics: Variation in study selection preferences across political vs. administrative roles, government levels, policy areas, geographical area, and demographic characteristics.
- Correlations between choices and other survey measures: Associations between revealed study characteristic preferences and evidence use patterns, attitudes toward evidence consultation, and attitudes toward science-policy collaboration.
- Preference class identification: Distinct preference segments for study characteristics, including characterization of segment membership based on participant characteristics.

Design.
This study employs a discrete choice experiment (DCE) with a mixed randomized controlled design. The experiment uses a 2 arms between subject treatment (institutional context: collaborative vs conflicting), and the following withing subject varying attributes levels: 5 (geographic coverage levels), (time period levels), 2 (direct experience collection: yes vs no), 3 (impact evaluation type levels).

Scenario description.
All participants are exposed to the same vignette (hypothetical scenario): they are responsible for developing a policy proposal within a three-month deadline addressing a highly relevant issue in their area of competence. They need to select which types of scientific studies should be prioritized for inclusion in an evidence synthesis to support their policy development. Participants are asked to imagine a specific policy problem consistent with the described scenario.

Choice task structure.
Participants complete 8 binary choice tasks, each presenting two alternative types of scientific studies for inclusion in the evidence synthesis. The two study options are presented side-by-side on desktop computers or vertically stacked on mobile devices, with attributes displayed using visual icons and standardized descriptions. In each task, participants must select their preferred study type to include in their evidence synthesis by clicking the corresponding button. Choice tasks are presented one at a time and are self-paced, but must be completed in a single session. Participants cannot return to previous tasks once submitted.

Attribute specification.
Each study type is characterized by four attributes with the following levels:
1. Geographic coverage (two levels, Italian and non-Italian, with the values of non-Italian varying for realism):
• Italy
• Southern Europe (Spain, Greece, Portugal)
• Nordic countries (Sweden, Norway, Denmark)
• Central Europe (Germany, Austria, Switzerland)
• Anglo-Saxon countries (United Kingdom, United States, Canada)
2. Time period analyzed:
• 1990-2000
• 2000-2010
• 2015-2025
3. Collection of direct experiences:
• Yes: the study collects direct experiences from beneficiaries and program managers through in-depth interviews
• No: the study does not collect direct experiences through interviews
4. Type of impact evaluation:
• Before and After Design
• Quasi Experimental Design
• Experimental design

Treatment manipulation.
Participants are randomly assigned to one of two between-subject conditions that differ in the institutional climate:
• Treatment group (conflicting climate): Commission members strongly support different policy solutions and show little openness to alternative approaches;
• Control group (collaborative climate): Commission members favor different policy solutions but are open to considering alternative approaches

The choice set are generated through a fully randomized design. In each choice set,the values of the four attributes are allowed to vary randomly in each option of each choice set. The only constraint is that the two options cannot be identical.

The data collection will be stagered, we will invite participants gradually, spanning the invitations across several weeks. If the response rate for the initial 5000 emails sent is lower than 3%, we will use the collected data to generate priors and change the design to a D-efficient fractional design, to ensure a higher efficiency for the estimation of the main effects. The intial pool of respondents will be assigned to a separate block and we will constrol for it in the analysis.

Randomization is implemented through oTree's built-in randomization functions.

The randomization unit is the individual participant. Each participant is independently randomly assigned to one of the two treatment conditions upon entering the oTree session.

No

Each individual policymaker is a unit of analysis. The target is 1,000-1,300 respondents, which corresponds to an 8-10% response rate from the initial 13,000 contact list.

1,000-1,300 respondents, who are asked to perform 8 choice-tasks each.

Equal randomization to treatment/control groups: 500-650 policymakers in treatment group, 500-650 policymakers in control group