Non-Take Up in Social Programs: Evidence from a Randomized Controlled Trial

This RCT will evaluate the effectiveness of two different treatments to address NTU:
1. In-person Intervention: there will be a number of outreach in selected neighborhoods which include awareness activities of the benefit as well as searching for potential beneficiaries of the benefit. There will also be advisory services to assist with the application of the benefit.
2. Social Network Campaign: there will be a campaign through Facebook and Instragram to raise awareness of the benefit and to encourage applications.

2023-04-10

2023-06-30

Our main outcomes will be the number of new applications to the benefit measured at the aggregate (node) level, the general opinion of residents about the benefit, and the level of trust in institutions. The last two outcomes would be examined at the individual level on the anonymized dataset.

The description of these outcomes, hypothesis and data source can be found in the Pre-Analysis Plan.

• Financial situation: income, “capacity to pay bills by the end of the month”.
• Knowledge and opinion of the benefit.
• Trust in public administrations, and capacity of political institutions to address citizens’ main problems.

The unit of randomization is the census tract ( which we refer as the node). Each of the 400 nodes was randomly assigned to one of the following groups: (1) group 1 (G1 from now on), receiving in-person intervention (n=100); (2) group 2 (G2), receiving social media intervention through Facebook and Instagram (n=100); (3) group 3 (G3), receiving both interventions (i.e., in-person and via social media) (n=100); and group 4 (GC), receiving none (pure control group) (n=100). In addition to this, nodes assigned to the social media intervention (G2 and G3) are then randomly assigned to one of 2 groups (GA or GB), that differ in the order they will receive each of the 3 social media ads.

The units of analysis in this RCT are census tracts (secciones censales) in Spain. As a first step to select which census tracts would participate in the experiment, EAPN-ES chose 510 census tracts with high shares of low-income population and that had the appropriate facilities and network of collaborators for the intervention to operate. To verify the first criterion, the partner used information from the Spanish National Statistics Institute regarding the percentage of households with income below 40% of the median income residing in each census tract (40% of the median household median income corresponds to approximately to 535 euros a month in 2020). To verify the second criterion, the partner consulted through their network of NGOs if they had a local NGO or organization that could lend appropriate facilities to conduct the intervention. In particular, some indoor private office space to help with the applications was necessary. Note that, while the presence of local organizations was important, the treatment was provided in a centralized manner by the partner organization EAPN-ES. We provide more details on this below.
Those facilities were referred as “contact point”, and they are typically within the census tract of study or near them. A contact point could be, for example, a social center, a day center, a soup kitchen, an association, a reception center, or a homeless shelter. In this project, each census tract selected is called a “node” and each contact point is the “point”. As will be explained in the description of the interventions, the node becomes the relevant unit for the outreach campaign, as it is where posters will be placed, and flyers distributed. However, the counseling and support for applying to the benefit was conducted at the point. In the case of the social media intervention, the treatment is done at the zip code level corresponding to the node.
The research team in collaboration with the Ministry did the second step to select 400 nodes out of the initial 510 nodes proposed by the partner. We visualized the nodes and points in a map and eliminated from the initial proposal those that were in proximity in order to minimize the risk of contamination. Finally, EAPN-ES further eliminated some additional nodes whose distribution across the regions was complicated for logistical reasons. The final sample of 400 nodes is at least 300 meters of distance from any other node and they do not share the same postal code as any other node.

Stratified randomization at the neighborhood level. See the document of the Pre-Analysis Plan for more details.

Randomization was done at the node (i.e., census tract) level.

Yes

400 census tracts (nodes)

12,000 respondents of surveys and 400 observations for aggregate-level statistics from administrative data.

100 nodes by each of the 4 study arms

The Principal investigators conducted power calculations on two outcomes: total number of applications and satisfaction with the benefit. For the first outcome, we considered the standardized variable, so that we assumed a control group’s mean of 0 and standard deviation (SD) of 1. For the second variable, we used the replies to the question “What is your opinion about the IMV?”, collected during the second wave of surveys done by Martinez-Bravo and Sanz (2021) in November 2020. The scale of answers go from 0 (“I totally dislike it”) to 10 (“I totally like it”). We considered a mean satisfaction for the control group of 5.83 and SD of 3.03. In addition to these main assumptions, we considered a significance level of 0.05, an intra-cluster correlation of 0.5, 100 clusters per treatment arm, and an average cluster size of 30. In the 1x1 comparisons (e.g., when trying to estimate the impact of in-person or on-line interventions comparing outcomes with that of the control group), we will be powered at the 80% to detect a minimum effect of 0.28 SD. In the case of satisfaction, we would be able to detect effects of 0.86 or larger (equivalent to a treatment group satisfaction of 6.69 or larger). In the specification where we combine our three treatment groups into a single group, we considered for power calculations 100 clusters in the control group, and 300 clusters in the treatment group. In this case, still considering a power of 80%, we would be able to detect a minimum effect of 0.23 SD of the number of applications, and a minimum change in the level of satisfaction of 0.7. It is important to note that these calculations are conservative since they do not take into account the fact that randomization was stratified. In addition to this, in the diff-in-diff and ANCOVA specifications, we will consider information from the baseline survey which will allow to increase power (or, equivalently, reduce the minimum detectable effects for the given power).