Belief formation, signal quality and information sources: Experimental evidence on air quality from Pakistan

In our intervention, we send air quality forecasts via SMS to a sample of Lahore residents in a working class neighborhood. We developed an ensemble forecast model of day-ahead air pollution using data inputs from multiple sources, including government and private monitors. We then experimentally vary the salience of information sources when we give forecasts to the participants, holding constant the actual forecast. In one arm, respondents are told that the forecasts are constructed using data from Punjab Environmental Protection Department (EPD), a government agency responsible for reporting on air quality. In the other arm, respondents are told that the forecasts are constructed using data from a citizens’ group called Pakistan Air Quality Initiative (PAQI). We identify treatment effects on measures of their beliefs over air quality and preferences via incentive-compatible elicitation.

2023-02-16

2023-06-30

1. Demand for air quality information as the willingness-to-pay (WTP) for SMS-based air quality forecasts
2. Beliefs about air quality levels as the absolute error of incentivized t + 1 forecast of PM2.5 concentration
3. Perceived accuracy of air-quality information source as the absolute error of incentivized guess of the SMS’s forecast
4. Preference for information source as the share of donations to government vs. citizens’ group

1. Demand for air quality information as the willingness-to-pay (WTP) for SMS-based air quality forecasts
• The outcome is defined as the amount respondents’ are willing to pay in PKR. We elicit respondents’ willingness to pay for the SMS forecast using the Becker- DeGroot-Marshak (BDM) method (Becker et al. 1964). In the endline survey, we ask for the respondents’ willingness to pay for the SMS-based air quality forecast messages. They have been receiving these messages for the past three months and we ask for their willingness to pay for an additional two months. In the prompt, we make the experimentally assigned source salient by reminding them that the forecast is built using data from the said source. The bid’s ceiling is set at PKR 400.
2. Beliefs about air quality levels as the absolute error of incentivized t + 1 forecast of PM2.5 concentration
• The outcome is defined as the absolute difference between the actual PM2.5 concentration and the respondent’s forecast, divided by the actual PM2.5 concentration. In both baseline and endline surveys, we ask respondents to make an incentivized guess of the air pollution level on day t + 1. In the baseline survey, we show respondents a table containing the average, minimum, and maximum of the average daily PM2.5 concentration over the last calendar week. We then ask them to forecast tomorrow’s average PM2.5 concentration. Respondents receive PKR 250 if their guess falls within 5% of the actual levels, PKR 150 if within 10%, and PKR 50 if within 20%. In the endline, we first elicit the forecast without the table containing the information from the previous calendar week. We then allow the respondents to revise their forecast after showing them the table.
3. Perceived accuracy of air-quality information source as the absolute error of incentivized guess of the SMS’s forecast
• The outcome is defined as the absolute difference between the respondent’s guess of the PM2.5 forecast generated by our model and their own forecast for t + 1. In the endline survey, we not only ask respondents to forecast the actual PM2.5 concentration for tomorrow, but also the value of our SMS forecast. The guess is financially incentivized, as in the guess for the actual PM2.5 concentration for tomorrow.
4. Preference for information source as the share of donations to government vs. citizens’ group
• The outcome is defined as the share of PKR 100 donated to a government agency for an environmental cause, as opposed to the citizen’s group. We offer an opportunity to donate PKR 100 between two sources for environment protection purposes: a government institution and PAQI.

All secondary outcomes are listed in the pre-analysis plan, which will be made publicly available after the conclusion of our trial.

We study how citizens update their beliefs about air quality and modify their behaviors, and how that depends on the information source. We also study how information quality affects citizens’ beliefs about the reliability of, and preference for, government vs. private sources. We will run a field experiment in which we provide a day-ahead pollution fore- cast via SMS to lower-middle income citizens in Lahore. We construct ensemble forecast models taking inputs from both government and private sources. We experimentally vary to which source the information is attributed (Government or Private). We collect incentivized forecast measures of air pollution before and after the intervention to track the evolution of citizens’ beliefs about air quality. We conduct lab-in-the-field games to identify citizens’ trust and preferences for the two sources, as well as their willingness to pay for the forecast service, which we provide for free during the experimental phase. We also measure impact on avoidance behavior from poor air quality by collecting time-use data and reported uses of high-filtration masks. Further detail will be made available in the pre-analysis plan after the conclusion of our trial.

We conducted the blocking and randomization on R using blockTools, a package that al- lows us to block observations on a large number of covariates and can accommodate continuous variables without discretizing them. We use the optimal-greedy algorithm and generate blocks using the Minimum Volume Elipsoid (MVE) estimator. We are primarily concerned about balance on outcome variables at baseline, as well as the “take-up” in terms of exposure and comprehension of our SMS forecast messages.

individual

No

1010

1010

505 for government (EPD) arm, 505 for private (PAQI) arm

We estimate the minimum detectable effect sizes on the five hypotheses that we pre-specify in the pre-analysis plan, using new data from the baseline survey when available. The outcomes, hypotheses, sample means, and standard deviations are: 1. Willingness-to-pay (WTP) for SMS-based air quality forecasts is greater than 0 regardless of the source to which the information is attributed: 89.6 (45.2) 2. Willingness-to-pay (WTP) for SMS-based air quality forecasts is differentially affected by treatment: 89.6 (45.2) 3. Absolute error of incentivized t + 1 forecast of PM2.5 concentration, divided by the truth, is differentially affected by treatment: 0.72 (0.42) 4. Perceived accuracy of air-quality information source as the absolute error of incentivized guess of the SMS’s forecast is differentially affected by treatment: N/A 5. the amount out of PKR 100 donated to a government agency for an environmental cause, as opposed to the citizen’s group, is differentially affected by treatment: 50.1 (15.0) For hypotheses 1. and 2., we use the sample statistics from Ahmad et al. (2022) as we do not collect these outcomes in the baseline of this study. We do not have relevant statistics available from either the baseline or from Ahmad et al. (2022) for hypothesis 3., but we expect the outcome variable for it to have a similar distribution to the one for hypothesis 3.. We find that we are able to detect impacts of 0.43 standard deviations, which equals to PKR 19.4 in the willingness to pay (for hypothesis 2.), 0.18 for hypothesis 3., and 6.4 for hypothesis 5.. For the test of means for hypothesis 1., we find that we are powered to detect that willingness to pay is greater than PKR 3.6. Although the minimum detectable impact is fairly large relative to the standard deviation in this second scenario, the treatment effect sizes are relatively small in the outcomes’ units. Furthermore, there are several reasons why our assumptions may not hold or statistical precision could be improved. First, we hope to reduce standard errors by including controls selected via a double-post-selection method using LASSO. Assuming a 30-percent reduction in standard errors, the minimum detectable effects would be 0.30 standard deviations. Second, the willingness-to-pay statistic from Ahmad et al. (2022) may be outdated after two years of high inflation.

PAP: 28 May 2023