Supporting Pathways Out of Poverty: A Randomized Evaluation of AMP Up Boston

AMP Up Boston provides personal assistance through Mobility Mentoring, a system of goal setting, incentives, and individualized coaching designed to holistically address the problems participants face. Through regular meetings, mentors assess and track progress along five domains – family life, health, finances, education, and career. Mentors work with families to develop a customized plan to achieve economic independence, broken down into small, manageable steps. This involves setting goals to pay off debt, obtain additional schooling, apply for jobs, access healthcare, and obtain stable housing. Additionally, mentors offer targeted financial incentives to help encourage individuals to save, obtain training towards a new career, or take other steps toward the goals they have set for themselves. The AMP Up Boston program lasts three years.

2021-10-27

2025-10-27

Our primary outcome is annual labor earnings (or quarterly earnings, if available).

Our secondary outcomes are:
- employment (an indicator for whether an individual is employed)
- high-wage employment (an indicator for whether an individual is employed in a high-wage job)
- total annual household income
- housing stability and housing support (receipt of rent subsidy, amount of rent subsidy, homelessness, neighborhood poverty rate)
- personal finance (credit score, overdue debts, delinquencies) from credit bureau data
- education and training (completion of high school diploma/GED/HiSET, completion of certification, completion of associate's degree, completion of bachelor's degree, training and employment program participation)
- health and well-begin (survey measures of self-assessed physical and mental health if follow-up survey funding available)
- public assistance (receipt and amount of SNAP and TANF benefits)

Subgroup Analyses
We plan to examine heterogeneity of effects on the listed primary and secondary outcomes by:
- race
- primary language
- employment status at start of study

The study will recruit participants for the AMP Up Boston program from among eligible residents in BHA public housing and recipients of BHA housing vouchers. Since more residents are likely to be interested in Mobility Mentoring than EMPath can serve, we will randomize access to the program among interested eligible residents. This will create two similar groups, one which is offered Mobility Mentoring through AMP Up Boston (the treatment group) and one which is not (the control group). We can then compare outcomes between the two groups to see what difference Mobility Mentoring makes for those who receive the program, on top of the gains they would have made on their own as seen in the control group. AMP Up Boston participants will be able to receive three years of Mobility Mentoring Services. We plan to follow study participants’ outcomes for ten years from random assignment through administrative data, allowing us to see whether recipients of Mobility Mentoring make temporary, short-term gains or sustainable, long-term changes in their economic situation.

Not available

Computer random number generator

Individual

No

N/A

570. We will end recruitment once we have 200 active program participants in the treatment arm, so this number may be smaller or larger depending on the take-up rate of the program among the treatment group. Also, if funding can be attained to hire additional mentors and serve more participants, we will try to increase the sample size to 267 active program participants and increase the treatment and control groups (with a 50 percent treatment randomization rate) to reach this number receiving Mobility Mentoring Services.

285 treatment participants, 285 control participants. We will end recruitment once we have 200 active program participants in the treatment arm, so this number may be smaller or larger depending on the take-up rate of the program among the treatment group. Also, if funding can be attained to hire additional mentors and serve more participants, we will try to increase the sample size to 267 active program participants and increase the treatment and control groups (with a 50 percent treatment randomization rate) to reach this number receiving Mobility Mentoring Services. Finally, if the take-up rate of services among the treatment group is below 70%, we may continue recruitment until we achieve 80% power to detect the minimum detectable effects noted in our power calculations.

Using historical data on employment and earnings for previous Mobility Mentoring enrollees, we estimate the minimum detectable effect (MDE) among the treated (i.e. treatment-on-treated (TOT)) necessary to observe a treatment effect with 5% statistical significance at 80% power. We assume a sample size of 580 study participants, 50% of whom are randomly assigned to the treatment group and offered the opportunity to enroll. We further assume an effective take-up rate of 70%, meaning that the difference between the treatment and control group participation rate is 70%. (e.g. 75% in the treatment group vs. 5% in the control group). Finally, we assume that controlling for the lagged dependent variable soaks up 20% of outcome variance. These calculations suggest that this study has sufficient power to detect a 20% or 14 percentage point increase in employment rates and a 34% or $4,180 increase in annual earnings. Provided additional funding, the expanded sample that would increase the sample size of active program participants to 267 has power to detect a 17% or 12 percentage point increase in employment rates and a 29% or $3,609 increase in annual earnings. These gains are within the range of observational changes in income and employment; 97% of graduates of EMPath’s flagship program were employed at exit relative to 65% at entry. Graduates also saw their annual incomes rise by 96% from $23,000 to $45,000 on average. These gains, however, may be an artifact of low-achieving participants disproportionately attritting from the sample or regression to the mean (if participants enroll during acute economic hardship). RCTs of intensive job training and employment programs, such as Job Corps, Project QUEST, Year Up, and the Sectoral Employment Impact Study (SEIS), provide information on possible treatment effect ranges as well as control-group trajectories absent treatment (Katz, Roth, Hendra, and Schaberg 2020). Treatment effects on earnings at several years out range between 8.1% (Job Corps) and 30 to 40% (Year UP) and employment effects range between 3.5% (Job Corps) and 20.3% (SEIS). Therefore, the present study is powered to detect treatment effects of effective employment programs, such as SEIS and Year Up. Given that Mobility Mentoring offers more comprehensive support services than job training and employment programs, we view the treatment effects of effective employment programs as a plausible lower bound on the range of policy-relevant treatment effects for Mobility Mentoring. The earnings gains observed among low-income single-parent households in the control groups of Project QUEST and SEIS studies were from 38 to 55%. This study therefore still has sufficient power to detect observed earnings gains even if a substantial portion of pre/post gains observed for past participants resulted from regression to the mean (i.e., 96%- 55% = 41%). References: Katz, Lawrence F., Jonathan Roth, Richard Hendra, and Kelsey Schaberg. 2020. “Why Do Sectoral Employment Programs Work? Evdence from WorkAdvance.” NBER WP No. 28248.