Non-Take Up in Social Programs: Evidence from a Randomized Controlled Trial

This RCT will evaluate the effectiveness of two different treatments to address NTU:
1. In-person Intervention: there will be a number of outreach in selected neighborhoods which include awareness activities of the benefit as well as searching for potential beneficiaries of the benefit. There will also be advisory services to assist with the application of the benefit.
2. Social Network Campaign: there will be a campaign through Facebook and Instragram to raise awareness of the benefit and to encourage applications.

In-person Intervention: During a period of 10 weeks there will be a number of in-person activities in the nodes selected to receive this treatment (G1 and G3). At least one field officer is assigned to each node. Additionally, every 3 or 4 nodes there is a coordinator supervising the activities. All local teams are coordinated by the central team of EAPN-ES. The activities that the local teams implement are the following. Outreach: Activities to raise awareness about the benefit, inform and identify potential beneficiaries of the benefit who are in a situation of NTU. Advisory services: These activities include providing information, guidance, and advice to individuals who are potentially eligible for NTU and are part of the project. It also involves conducting initial assessments to verify the eligibility of potential beneficiaries and addressing other issues related to NTU. Support: Assisting in obtaining necessary administrative documentation through guidance and coordination. Guidance: Activities aimed at providing individuals with information about relevant institutions and government agencies where they can complete the necessary procedures for obtaining documentation related to applications to the benefit. The partner will also offer guidance on where to submit applications to the benefit and help to apply for it. Coordination: Actions aimed at coordinating and mediating between the individual and the administration, typically carried out by the local field technician. This includes coordinating with Social Services, for example.
Social Media Intervention: A social adds campaign will be launched through the company Meta, which delivers adds in Facebook and Instagram. The adds appear in Facebook’s Feed or in the right-hand-side column in Facebook. In Instagram they appear as Feed or in Stories. In both platforms they appear as “promoted”, signaling that they are part of an advertisement.
These adds are shown only to individuals residing in selected zip codes. The company Meta infers residence, by the location where residents spend most nights.
There are three different types of adds:
1. The first message combines photos and the following text: “If you don’t have enough for the basics. If you struggle to make ends meet at the beginning of the month. If you have difficulties filling your grocery cart or providing for your family’s needs… This benefit is for you: Minimum Income Scheme”.
The add also provides a link to the Social Security website which contains a simulator in which potential applicants can self-assess their eligibility.
2. The second add consists of a video which shows the following text: “Did you know you can apply for the IMV even if you are employed? Or that you can receive it along with other benefits? Are you aware of the child supplement for the IMV? It aims to provide additional support to individuals with children. Access all the information about the IMV and check if you meet the requirements. Because this is truly vital.”
The add also provides a link to the Social Security Website that provides information on the benefit.
3. The third message combines image and messages. It aims to reduce stigma by emphasizing that many households are already recipients of the benefit. It provides the following message: “Over 1,500,000 people are already receiving the IMV. This assistance can also be for you. If you lack the basics, if you struggle to make ends meet: claim what is rightfully yours.”
The add also provides a link to the application website of the benefit.
The social add campaign will last 10 weeks. These weeks are divided in three stages: stage 1, weeks 1-3; stage 2 weeks 4-7; stage 3 weeks 8-10. Adds 1, 2 and 3 alternated in time across stages.
Group A receives messages across the three stages in the following order: Add 1; Add 2; Add 3.
Group B receives messages across the three stages in the following: Add 2; Add 3; Add 1.
Note that the implementing partner for all treatments is EAPN. They may collaborate with public or third sector organizations but their tasks are not delegated, and EAPN remains responsible for the intervention.

2023-04-10

2023-06-30

Our main outcomes will be the number of new applications to the benefit measured at the aggregate (node) level, the general opinion of residents about the benefit, and the level of trust in institutions. The last two outcomes would be examined at the individual level on the anonymized dataset.

The description of these outcomes, hypothesis and data source can be found in the Pre-Analysis Plan.

• Financial situation: income, “capacity to pay bills by the end of the month”.
• Knowledge and opinion of the benefit.
• Trust in public administrations, and capacity of political institutions to address citizens’ main problems.

The unit of randomization is the census tract ( which we refer as the node). Each of the 400 nodes was randomly assigned to one of the following groups: (1) group 1 (G1 from now on), receiving in-person intervention (n=100); (2) group 2 (G2), receiving social media intervention through Facebook and Instagram (n=100); (3) group 3 (G3), receiving both interventions (i.e., in-person and via social media) (n=100); and group 4 (GC), receiving none (pure control group) (n=100). In addition to this, nodes assigned to the social media intervention (G2 and G3) are then randomly assigned to one of 2 groups (GA or GB), that differ in the order they will receive each of the 3 social media ads.

The units of analysis in this RCT are census tracts (secciones censales) in Spain. As a first step to select which census tracts would participate in the experiment, EAPN-ES chose 510 census tracts with high shares of low-income population and that had the appropriate facilities and network of collaborators for the intervention to operate. To verify the first criterion, the partner used information from the Spanish National Statistics Institute regarding the percentage of households with income below 40% of the median income residing in each census tract (40% of the median household median income corresponds to approximately to 535 euros a month in 2020). To verify the second criterion, the partner consulted through their network of NGOs if they had a local NGO or organization that could lend appropriate facilities to conduct the intervention. In particular, some indoor private office space to help with the applications was necessary. Note that, while the presence of local organizations was important, the treatment was provided in a centralized manner by the partner organization EAPN-ES. We provide more details on this below.
Those facilities were referred as “contact point”, and they are typically within the census tract of study or near them. A contact point could be, for example, a social center, a day center, a soup kitchen, an association, a reception center, or a homeless shelter. In this project, each census tract selected is called a “node” and each contact point is the “point”. As will be explained in the description of the interventions, the node becomes the relevant unit for the outreach campaign, as it is where posters will be placed, and flyers distributed. However, the counseling and support for applying to the benefit was conducted at the point. In the case of the social media intervention, the treatment is done at the zip code level corresponding to the node.
The research team in collaboration with the Ministry did the second step to select 400 nodes out of the initial 510 nodes proposed by the partner. We visualized the nodes and points in a map and eliminated from the initial proposal those that were in proximity in order to minimize the risk of contamination. Finally, EAPN-ES further eliminated some additional nodes whose distribution across the regions was complicated for logistical reasons. The final sample of 400 nodes is at least 300 meters of distance from any other node and they do not share the same postal code as any other node.

Stratified randomization at the neighborhood level. See the document of the Pre-Analysis Plan for more details.

Randomization was done at the node (i.e., census tract) level.

Yes

400 census tracts (nodes)

12,000 respondents of surveys and 400 observations for aggregate-level statistics from administrative data.

100 nodes by each of the 4 study arms

The Principal investigators conducted power calculations on two outcomes: total number of applications and satisfaction with the benefit. For the first outcome, we considered the standardized variable, so that we assumed a control group’s mean of 0 and standard deviation (SD) of 1. For the second variable, we used the replies to the question “What is your opinion about the IMV?”, collected during the second wave of surveys done by Martinez-Bravo and Sanz (2021) in November 2020. The scale of answers go from 0 (“I totally dislike it”) to 10 (“I totally like it”). We considered a mean satisfaction for the control group of 5.83 and SD of 3.03. In addition to these main assumptions, we considered a significance level of 0.05, an intra-cluster correlation of 0.5, 100 clusters per treatment arm, and an average cluster size of 30. In the 1x1 comparisons (e.g., when trying to estimate the impact of in-person or on-line interventions comparing outcomes with that of the control group), we will be powered at the 80% to detect a minimum effect of 0.28 SD. In the case of satisfaction, we would be able to detect effects of 0.86 or larger (equivalent to a treatment group satisfaction of 6.69 or larger). In the specification where we combine our three treatment groups into a single group, we considered for power calculations 100 clusters in the control group, and 300 clusters in the treatment group. In this case, still considering a power of 80%, we would be able to detect a minimum effect of 0.23 SD of the number of applications, and a minimum change in the level of satisfaction of 0.7. It is important to note that these calculations are conservative since they do not take into account the fact that randomization was stratified. In addition to this, in the diff-in-diff and ANCOVA specifications, we will consider information from the baseline survey which will allow to increase power (or, equivalently, reduce the minimum detectable effects for the given power).