Experimental Design
Blocking:
We use the answers to the baseline survey to sort participants into homogenous blocks. The 28 variables used are listed in column 1 of Table 1 (in the attached pre-analysis plan). Appendix A provides further details on the definition of these variables.
Pairwise distances between observations are calculated using the Mahalanobis distance.4 We construct blocks containing 32 observations each. The blocks are chosen to minimize the total sum of distances between pairs of observations within blocks. We do so using the R package blockTools (Moore and Schnakenberg, 2016). We then discard all blocks with a maximum within- block distance greater than 14 (to avoid poorly matched observations), as well as one block with less than 32 observations.
Random assignment within blocks:
Within each block, treatment is assigned uniformly at random. We assign 2 out of the 32 observations in a block to the treatment group, 26 observations to the control group, and the remaining 4 observations to a “reserve,” which is to be sampled in case of attrition of observations from the treatment or control group.
These numbers are chosen based on the following considerations: We want two treated units per block, in order to be able to calculate standard errors for the sample average treatment effect; cf. Athey and Imbens (2017) and our discussion of inference below. We don’t want more treated units per block, to keep blocks as homogenous as possible. The budget constraints of our implementation partner are furthermore such that we can survey 13 control units for every treated individual.
Lastly, because we have 107 treated individuals in total (an odd number), one additional individual from one block is chosen at random to participate in the treatment.
Weighted sampling of blocks:
This procedure results in 273 blocks, while our project budget allows for 53 blocks. These blocks are furthermore not fully representative for the baseline sample, because not all individuals who were invited to participate in the baseline survey passed eligibility and had non-missing responses in the questions we used for blocking (see above) and because of our discarding of poorly matched blocks.
In order to obtain a representative sample of blocks, we create block level sampling weights. These weights are chosen so as to match the distribution of gender, education groups, and income groups of eligible participants in the screening survey. We then draw a sample of 53 blocks from the 273 available blocks using these sampling weights, to obtain a representative subsample.
This results in 107 individuals assigned to treatment, 1377 assigned to the control group, and 212 individuals assigned to the “reserve,” distributed evenly across 53 blocks.