Minimum detectable effect size for main outcomes (accounting for sample
design and clustering)
Applications to participate in the project will be voluntary, so some degree of uncertainty remains
regarding the number of applicants the project will recruit into the study’s sampling frame and,
in turn, the size of the initial sample for randomization. We will work closely with the
TechnoServe team to ensure an effective marketing campaign is implemented – with clear and
consistent communication that attracts many green entrepreneurs across India to apply online.
If TechnoServe can reach a broad audience through its marketing campaign, then the number of
recruited firms that meet the eligibility criteria (sampling frame of ~630 firms) should be
sufficient for obtaining the sample size required based on our power calculations (initial sample
of ~400 firms at randomization). This is a nontrivial but critical task. Having fewer than 133 firms
per experimental group would threaten statistical validity by reducing the power of our overall
study design. We will advise the TechnoServe team and coordinate recruitment efforts to give
our impact evaluation study the best chance of a successful launch.
In addition, low take-up rates also constitute a typical challenge in multi-year field experiments
with small firms. The upfront adoption of a program, as well as continued compliance, play an
important role in maintaining a high level of intervention strength (i.e., a ‘strong hammer’ that
can move the needle in terms changing entrepreneur behaviors and business activities). A
stronger intervention should improve the signal-to-noise ratio by increasing the ‘signal’ or effect
size (i.e., a larger treatment coefficient in our regressions). Achieving a minimum take-up rate of
90% is therefore essential for having the statistical power needed to interpret the results of our
analyses. Offering attractive programs (in all three stages) is helpful in this regard. The funneling
approach is also designed to encourage take up among the screened and targeted firms.
Moreover, take-up rates could also be enhanced by the fact that all firms in the project (including
those outside the funneling treatment group) may be offered the chance to obtain green financial
capital via project-supported access to carbon markets at the end of the study period.
Statistical power can also be enhanced through better measurement of outcomes that improves
the signal-to-noise ratio by decreasing ‘noise’ or variance (i.e., smaller standard errors in our
regressions). In addition to conducting numerous data checks (especially for outcome measures),
we will also reduce the influence of outliers by winsorizing values 1% on each tail and
constructing indexes to represent a family of outcomes. For instance, we can create an overall
‘performance index’ that captures each firm’s baseline-to-endline change in sales, profits and
employees. Likewise, we can construct an overall ‘carbon index’ that represents an individual
firm’s greenhouse gas emissions via operations, offerings and offsets. Our data collection
instruments can also be designed to incorporate anchoring, aggregating and adjusting steps that
improve measurement of key outcomes (see Anderson, Lazicky and Zia 2021).
Sample Size Calculations. The sample size is determined by the TechnoServe program budget, as
well as the research team’s recognition that it may be challenging to secure a sample of more
than 400 green SGBs in India that meet all eligibility criteria and provide complete data on the
Recruitment Survey and Baseline Survey. Our power calculations are then designed to check
whether this is a reasonable sample size for detecting effects on changes in economic and
environmental outcomes (e.g., firm performance, carbon emissions). Table 1 summarizes the
results of this exercise under different assumptions.
Given the high intervention intensity, on average, treated firms may achieve a 30% improvement
on a standardized index of firm performance (or carbon emissions). Next, under reasonable
assumptions – such as a coefficient of variation of 1.00, 80% power, and 2 post data collection
rounds – it follows that the study would be sufficiently powered if ~100 firms remained in each
experimental group for analysis (i.e., to compare the funnel treatment against control).
Obtaining this final sample is plausible if we start with an initial sample of ~133 firms per group:
• First, if we estimate a 10% non-adoption rate (or 90% take-up rate), then ~120 firms per
treatment group would be adopting their assigned program. This means ~13 firms would
no longer be participating in the study.
• Second, if we assume an additional 15% attrition in the post-intervention period, then
~18 firms per group would attrite (i.e., they cannot be located or do not respond in the
follow-up survey rounds).
24
IE Proposal
TechnoServe Green Accelerator
• In the end, our final sample size for analysis (via ANCOVA) would include ~102 firms per
experimental group. This is in line with the power calculations in Table 1 (refer to the
highlighted scenario with 1 pre and 2 post rounds5 of data collection).
Critically, however, if our design and budget allow for 3 post data collection rounds, the study will
have greater power (which decreases risk) and more degrees of freedom for analyzing
heterogeneous treatment effects (which adds insights).