Minimum detectable effect size for main outcomes (accounting for sample
design and clustering)
The sample size is determined by the project budget and geographic distribution of applicants (firms applying from other parts of Nigeria will not be offered the insourcing or outsourcing treatments, but will still be randomized to control, grants, training, or consulting, and to date we have 730 control, 434 training, and 434 consulting outside of Abuja and Lagos – we will use this other sample to measure impacts of these first two treatments in those regions).
Our power calculations are then designed to determine the minimum detectable effect possible with this sample. We are in a better position for making these calculations than most studies because the 2016 application window has closed and gives us data on the initial 943 experimental firms. Moreover, based on the first batches, we have over 90% take-up for the consulting, insourcing, and outsourcing treatments (training take-up data not yet available). We therefore assume 90% take-up in these calculations. We report the minimum detectable TOT effect assuming this take-up rate.
Table 2: Minimum Detectable Effect (80% power, 5% significance level)
Business Practices Score Employment Log Profits
Unit Level Firm Firm Firm
Variable Mean 6.31 4.22 13.05
Variable SD 0.75 2.59 1.35
MDE (in SD) 0.22 0.18 0.19
MDE (as % of reference mean) 2.6% 4.3% 25%
Business practices: calculation is made using only a single round of follow-up data. MDE for the ITT is then 0.15, so for the TOT is 1.1*0.15 = 0.165. This is 0.22 s.d., or 2.6% increase in the mean. Since the score is on a 10-point scale, this is equivalent to a 0.02 improvement in the proportion of practices implemented. This is less than half the size found in business training experiments cited in the literature review.
Employment: power calculations assume Ancova estimation, controlling for baseline value of employment. Autocorrelation is assumed to be 0.5. MDE is then 0.43 workers , so a TOT of 1.1*0.43 = 0.47 workers. This is 0.18 S.d., or a 4.3% increase on the control mean. For employment we believe the increase in the number of workers is the most useful metric. 0.47 workers increase in employment will occur if half the firms hiring an insourced worker keep this worker on, even if they do not grow enough to hire more workers. McKenzie (2017) finds high-growth enterprises achieve employment gains of 4-5 workers per firm from winning the business plan competition, so we are only trying to detect increases one-tenth of that size.
Log monthly sales: power calculations assume Ancova estimation, controlling for baseline value of sales. Autocorrelation is assumed to be 0.5. MDE for the ITT is 0.23, so TOT is 0.25 log points, or a 25% increase in the control mean. We aim to reduce this further through three steps: i) controlling for baseline covariates like sector, business practices, and employment size that can soak up some of the variation in size and reduce heterogeneity; ii) budget allowing, use multiple measurements of monthly sales within a given follow-up period as in McKenzie (2012); and iii) reduce measurement error and hence noise by utilizing a cross-checking survey technology.