Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Included in grant applications:
Since these firms are all operating in the same sector, selling similar products at the same scale, they will be more homogeneous than firms in many standard PSD interventions. This should improve the power of our experiment to detect impacts. Moreover, we believe that unit costs and travel time should be highly correlated over time, and will use multiple weeks of data to further improve power. The baseline data is required to fine-tune these power calculations, as we stand ready to adjust the design as needed if we find that based on the baseline data and initial take-up results that power is lower than anticipated. That said, here are some initial power calculations based on two outcomes that we have used to help guide our preliminary choices on sample size.
We assume take-up of the intervention will be 65% among those who express interest–this is conservative, given Agruppa have found approximately 80% take it up in their initial pilots. Our design at present is then random assignment at the market block level of 30 blocks to treatment and 30 to control, with each block containing 20 firms interested in the intervention, for a total of 600 treated and 600 control firms among the interested (we also have uninterested firms in each block).
Our starting point assumptions are that i) key outcomes are likely to be highly autocorrelated (firms that have long travel times today will have long travel times in a month due to geography, mode of transport, etc.); ii) key outcomes will have strong intra-cluster correlations in the cross-section within market blocks (e.g. firms that are all within the same block will charge similar prices and have similar travel times to market); and iii) the intra-cluster correlations will be much weaker in terms of changes (firms within blocks experience different shocks and react differently to the intervention).
To see this, consider the time in hours per week spent to travel to market. In the cross-section at time t, we model this for firm i in block b as:
Outcome 1: Reduction in travel time spent by firms to buy goods from the central market
Assumptions: Mean weekly travel time of 15 hours (estimated by Agruppa), standard deviation 5 hours, intra-cluster correlation of 0.6 (since firms within markets will have similar travel times). The intervention aims to reduce hours by an average of 5. The ITT is thus 3.25 hours (5*0.65).
Power using just a single round of follow-up data:
If randomization was at the individual level, power is 1:
sampsi 15 11.75, sd(5) n1(600) n2(600) gives power of 1
and we would need only 38 firms in each group to get 80% power
sampsi 15 11.75, sd(5) power(0.8)
But with an intra-cluster correlation of 0.6, and 20 firms per market, we need a minimum of 48 blocks (and hence 472 in each treatment group) to achieve 80 percent power.
Power using the baseline to improve power:
We assume now that using the baseline hours we have an autocorrelation of 0.7 with follow-up hours, so the residual variance becomes sqrt(1-0.72)*5 = 3.57 hours. We assume that this then reduces the intra-cluster correlation in hours to 0.3 (once the blockb component has been removed). Then we have:
sampsi 15 11.75, sd(3.75) power(0.8)
sampclus, obsclus(20) rho(.3)
Gives a minimum of 15 blocks, and 141 treated and 141 control in total needed.