Minimum detectable effect size for main outcomes (accounting for sample
design and clustering)
Using historical data on employment and earnings for previous Mobility Mentoring enrollees, we estimate the minimum detectable effect (MDE) among the treated (i.e. treatment-on-treated (TOT)) necessary to observe a treatment effect with 5% statistical significance at 80% power. We assume a sample size of 580 study participants, 50% of whom are randomly assigned to the treatment group and offered the opportunity to enroll. We further assume an effective take-up rate of 70%, meaning that the difference between the treatment and control group participation rate is 70%. (e.g. 75% in the treatment group vs. 5% in the control group). Finally, we assume that controlling for the lagged dependent variable soaks up 20% of outcome variance.
These calculations suggest that this study has sufficient power to detect a 20% or 14 percentage point increase in employment rates and a 34% or $4,180 increase in annual earnings. Provided additional funding, the expanded sample that would increase the sample size of active program participants to 267 has power to detect a 17% or 12 percentage point increase in employment rates and a 29% or $3,609 increase in annual earnings. These gains are within the range of observational changes in income and employment; 97% of graduates of EMPath’s flagship program were employed at exit relative to 65% at entry. Graduates also saw their annual incomes rise by 96% from $23,000 to $45,000 on average. These gains, however, may be an artifact of low-achieving participants disproportionately attritting from the sample or regression to the mean (if participants enroll during acute economic hardship).
RCTs of intensive job training and employment programs, such as Job Corps, Project QUEST, Year Up, and the Sectoral Employment Impact Study (SEIS), provide information on possible treatment effect ranges as well as control-group trajectories absent treatment (Katz, Roth, Hendra, and Schaberg 2020). Treatment effects on earnings at several years out range between 8.1% (Job Corps) and 30 to 40% (Year UP) and employment effects range between 3.5% (Job Corps) and 20.3% (SEIS). Therefore, the present study is powered to detect treatment effects of effective employment programs, such as SEIS and Year Up. Given that Mobility Mentoring offers more comprehensive support services than job training and employment programs, we view the treatment effects of effective employment programs as a plausible lower bound on the range of policy-relevant treatment effects for Mobility Mentoring. The earnings gains observed among low-income single-parent households in the control groups of Project QUEST and SEIS studies were from 38 to 55%. This study therefore still has sufficient power to detect observed earnings gains even if a substantial portion of pre/post gains observed for past participants resulted from regression to the mean (i.e., 96%- 55% = 41%).
Katz, Lawrence F., Jonathan Roth, Richard Hendra, and Kelsey Schaberg. 2020. “Why Do Sectoral Employment Programs Work? Evdence from WorkAdvance.” NBER WP No. 28248.