Experimental Design Details
The intervention is implemented using a randomized control trial approach to conduct a cross-sectional study on a population composed of individuals who are homogeneous in terms of geographical area and political orientations. Specifically:
- The intervention consists of administering an online survey where respondents are randomly exposed to different informational content about the official party platform.
- The population subject to the intervention is the entire membership of the Italian Democratic Party of Rome, totaling approximately 8,000 individuals.
The intervention is structured in four steps:
Step 1. We identify the official positions of the party as expressed by its leaders on a wide array of major political issues by reviewing various party official media platforms (e.g. the party website), other media platforms (radio, television, social networks, etc.), and other sources (e.g., the party manifesto).
Step 2. In the survey, respondents are asked to express their opinions on a sample of issues identified in Step 1. Respondents are exposed to different informational content through a randomized control trial approach. Members of the control group are asked to express their opinions without receiving any information on the positions of their leaders. Members of the treated group are asked the same questions but are informed about their leaders' positions by being exposed to quotes from them identified in Step 1. Within the treated group, the source of the quote used to report the leader’s opinion is randomized.
Step 3. The survey collection period is three months. During this time, we monitor potential changes in media coverage of political events to which respondents may be exposed while taking the survey, as these changes could affect their responses.
Step 4. We compare the answers provided by groups exposed to the direct treatment through the survey and the indirect treatment through media coverage changes. This analysis allows us to assess how different informational treatments impacted the responses of the surveyed members, either reinforcing or undermining their identification with the party.
Population is randomly assigned to treatment and control groups.
Participants in the treatment group are randomly divided into two subgroups: Group 1 and Group 2. Before answering a question about their political stance on a specific issue, both groups are presented with the party leadership’s position on that issue. However, the source of the information containing the party leadership’s position differs between the two groups.
In contrast, participants in the control group are not presented with the party leadership’s position before responding to the same question.
Assignment to the treatment and control groups is stratified to ensure that the personal characteristics of party members are balanced across groups. Similarly, stratification is applied when assigning participants to Group 1 and Group 2 within the treatment group. This process considers factors such as personal characteristics (e.g., gender and age) and the neighborhood of residence.
The three groups, treatment group 1, treatment group 2, and the control group, are all composed by the same number of party members, i.e. approximately 2,600 people.
During survey administration, we collect data on any sudden changes in media coverage of political events to identify participants exposed to different informational content before participating in the survey. This approach allows us to separate the potential confounding effect of media coverage from the intervention's impact and to further investigate how media coverage might influence the responses of surveyed party members. Data on media coverage is obtained using the Global Database of Events, Language, and Tone (GDELT) and other open-source databases, which apply natural language processing techniques to published media content to identify, on a daily basis: covered events, the number of media reports on an event, and the tone of the content covering the event.
All party members will be invited to participate in the survey at the same moment. An email will be sent to all of them through the party's mailing list. The party mailing list is owned and managed by the Italian Democratic Party of Rome, and it will not be accessible to the researchers. The email will be sent directly by the Italian Democratic Party of Rome. Researcher will have no access to the use of the mailing list to avoid any data transfer to the researchers. The email will be sent approximately one month after receiving the consent to conduct the study.
Participants will have three months to provide their answers. Invitations to participate in the study are sent via the party's mailing list; therefore, no exclusion criteria based on the individual characteristics of party members are applied, no external pressure is placed on members to participate, and no monetary incentive is offered for participation.
In the mailing list message, party members are informed that the party is interested in better understanding i) the political orientations of its members on a list of relevant issues and ii) which communication platforms are most effective in creating strong communication between the party leadership and its ranks, and for this reason the party has asked a team of researchers coordinated by Sapienza to conduct a study on its members. In the message, party members are further informed that the study results will be published in scientific publications and summarized in one or more reports for the internal use of party members. The message will also indicate that members willing to participate in the study should fill out an online survey to elicit their political orientations (a link to the survey will be included in the message, and the link will be different for treated and control group members). Details on the participants’ rights regarding participation and withdrawal will be provided via a link to the informed consent form, which explains that participants have the right to: i) choose not to participate in the study, ii) withdraw from the study at any time during the survey administration, and iii) withdraw from the study after the survey administration by requesting the deletion of their data without providing any explanation before data is completely anonymized (24 month from the conclusion of the data collection period).
The data collected through the survey is analyzed to understand the extent to which information influences respondents’ adherence to the party platform and whether respondents exposed to information react differently depending on the source of the informational content used. The goal is to study the causal relationship between i) the provision of information (directly through the survey or indirectly through media) and ii) changes in political orientations as recorded in the survey. Identification of this causal relationship is achieved by leveraging the randomization of the treatment (i.e., exposure to informational content).
The survey is administered using the software SurveyCTO. Question randomization is conducted using Stata, and data analysis is performed with Stata and R.