The Effect of Misinformation Training on Policymakers: Evidence from Pakistan

We will test the following hypotheses:

H1: Participants in the misinformation workshop will perceive misinformation as a more severe problem (H1a) and believe it is important to avoid spreading it (H1b).

H2: Participants in the misinformation workshop will better distinguish between false or misleading news and legitimate news in their behavioral sharing intentions (H2a) and accuracy ratings (H2b).

H3: Participants in the misinformation workshop will be less likely to select false or misleading news for inclusion in a government briefing.

H4: Participants in the misinformation workshop will be more likely to support government intervention to restrict false information online.

We will test the following hypotheses:

H1: Participants in the misinformation workshop will perceive misinformation as a more severe problem (H1a) and believe it is important to avoid spreading it (H1b).

H2: Participants in the misinformation workshop will better distinguish between false or misleading news and legitimate news in their behavioral sharing intentions (H2a) and accuracy ratings (H2b).

H3: Participants in the misinformation workshop will be less likely to select false or misleading news for inclusion in a government briefing.

H4: Participants in the misinformation workshop will be more likely to support government intervention to restrict false information online.

2022-07-05

2022-08-15

Measured Outcome variables

Prior to the treatment, we will measure the following:

… highest degree (indicator variables omitting bachelor’s)

… region or territory of origin (indicator variables omitting Punjab)

… years in government service (1=6-10 years…5=26+ years)

… political interest on a five-point scale where 1 = “Not at all interested” and 5 = “Extremely interested”

… conspiratorial predispositions on a five-point scale where 1 = “Strongly disagree” that “Much of our lives are being controlled by plots hatched in secret places” and 5=“Strongly agree”

… how much trust and confidence participants have in mass media (newspaper, TV, radio, and social media) on a 4 point scale where 1 = “Not at all” and 4 = “A great deal”.

… how frequently participants use Twitter on a 7-point scale where 1 = “Never” and 7 = “Daily”

… how frequently participants read political news on Twitter where 1 = “Never” and 7 = “Daily”

… how frequently participants tweet or retweet political news on Twitter where 1 = “Never” and 7 = “Daily”

… whether the participant or their staff uses or controls a Twitter account (0=none, 1=1 or more)

… political knowledge on a three-point scale measuring the number of correct answers to three knowledge questions

After the treatment, we will measure the following:

… treatment compliance where 1=in the treatment group and answered both questions abotu the presentation correctly and 0 otherwise

… recommending including a false or misleading news headline in a government briefing (where 1=selected a false or misleading headline for inclusion and 0 otherwise)

Indices

We will calculate the following mean values at the respondent level separately before and after the treatment:

… mean sharing intention for legitimate news (where 1 = “Extremely unlikely” and 6=“Extremely likely”)

… mean sharing intention for false or misleading news (where 1 = “Extremely unlikely” and 6=“Extremely likely”)

… mean difference in sharing intention between legitimate and false or misleading news (the difference of the means above calculated at the respondent level)

… a composite measure of support for government action to restrict misinformation (the mean response to “Governments should take steps to restrict false information online, even if it limits people from freely publishing or accessing information” where 1 = “Strongly disagree” and 5 = “Strongly agree” and “People’s freedom to publish and access information should be protected, even if it means false information can also be published” where 1 = “Strongly agree” and 5 = “Strongly disagree”; we will split these and analyze separately if they do not correlate at r=.6 or greater)

… a composite measure of the perceived importance of not sharing misinformation (the mean response to the perceived importance of only sharing accurate information where 1 = “Not at all important” and 5 = “Extremely important” and agreement that people in Pakistan’s Civil Service need to avoid spreading misinformation where 1 = “Strongly disagree” and 5 = “Strongly disagree”; we will split these and analyze separately if they do not correlate at r=.6 or greater)

… a composite measure of the perceived severity of misinformation (the mean response to the perceived seriousness of misinformation in Pakistan where 1 = “Not at all serious” and 5 = “Extremely serious”, self-reported concern about misinformation in Pakistan where 1 = “Not at all concerned” and 5 = “Extremely concerned”, and agreement that people in the civil service often spread misinformation where 1 = “Strongly disagree” and 5 = “Strongly agree”; we will split these and analyze separately if they do not clearly load on a single factor in an exploratory factor analysis)

We will calculate the following mean values at the respondent level after the treatment:

… mean perceived accuracy of legitimate news (where 1 = “Not at all accurate” and 4 = “Very accurate”

… mean sharing intention for false or misleading news (where 1 = “Not at all accurate” and 4 = “Very accurate”

… mean difference in perceived accuracy between legitimate and false or misleading news (the difference of the means above calculated at the respondent level).

Participants in each cohort of Pakistani civil servants (tax officers, less senior deputy ministers, and more senior deputy ministers) will be independently randomized with probability=.5 at the individual level by the researchers into a treatment or placebo group. Those in the treatment group will be assigned as part of their academy training to attend the treatment presentation and those in the placebo group will be assigned as part of their academy training to attend the placebo presentation.

We will compute treatment effects via OLS with robust standard errors. We will use a lasso variable selection procedure to determine the set of prognostic covariates to include in each model (i.e., separately for each model below). If one or more levels are selected from a factor variable, we will include only the selected level(s) in the model. Below is the set of candidate variables that we will select from (as measured pretreatment) -- if not otherwise stated, these variables will be treated as continuous:
-highest degree indicators
-region or territory of origin indicators
-years in government service
-political interest
-conspiratorial predispositions
-trust and confidence participants in mass media
-how frequently participants use Twitter
-how frequently participants read political news on Twitter
-how frequently participants tweet or retweet political news on Twitter
-whether the participant or their staff uses or controls a Twitter account (indicator)
-political knowledge

Our models will also include pretreatment measures of the outcome variable when available and fixed effects for the type of official (tax officer, less senior deputy ministers, more senior deputy ministers).

All models will be estimated on midline data collected starting the day after the intervention and on endline data collected approximately two weeks later.

To test our hypotheses, we will estimate the following models corresponding to the variables defined above:

H1a: misinformation_severity = B0 + B1*treatment + B2*misinformation_severity_pretreatment + official type fixed effects + lasso-selected controls

H1b: avoid_spreading_misinformation = B0 + B1*treatment + B2*avoid_spreading_misinformation_pretreatment + official type fixed effects + lasso-selected controls

H2a: sharing_discernment = B0 + B1*treatment + B2*sharing_discernment_pretreatment + official type fixed effects + lasso-selected controls

H2b: accuracy_discernment = B0 + B1*treatment + official type fixed effects + lasso-selected controls

H3: briefing_misinformation = B0 + B1*treatment + official type fixed effects + lasso-selected controls

H4: government_intervention = B0 + B1*treatment + official type fixed effects + lasso-selected controls

Participants in each cohort of Pakistani civil servants (tax officers, less senior deputy ministers, and more senior deputy ministers) will be independently randomized with probability=.5 at the individual level by the researchers into a treatment or placebo group. Those in the treatment group will be assigned as part of their academy training to attend the treatment presentation and those in the placebo group will be assigned as part of their academy training to attend the placebo presentation.

Randomization by computer

Individual civil servant

Yes

800

Sample size We expect to recruit approximately 800 participants but the exact total may vary. Sample size rationale This is the maximum sample size we can recruit.

400 per treatment arm (400 Misinformation education training and 400 placebo). Exact number may vary.

Pre-Analysis Plan Misinformation.pdf