Supporting Pathways Out of Poverty: A Randomized Evaluation of AMP Up Boston

Last registered on November 08, 2021

Pre-Trial

Trial Information

General Information

Title
Supporting Pathways Out of Poverty: A Randomized Evaluation of AMP Up Boston
RCT ID
AEARCTR-0008429
Initial registration date
October 22, 2021
Last updated
November 08, 2021, 10:34 AM EST

Locations

Primary Investigator

Affiliation
Harvard University

Other Primary Investigator(s)

PI Affiliation
Harvard University
PI Affiliation
Harvard University

Additional Trial Information

Status
In development
Start date
2021-10-27
End date
2032-10-27
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
Can Mobility Mentoring – a system of holistic, one-on-one mentorship combined with monetary incentives – help low-income residents achieve economic self-sufficiency? Specifically, we aim to examine the effects of Mobility Mentoring delivered through the AMP Up Boston program on participant economic outcomes (employment, earnings, household income, and financial health), housing stability, receipt of subsidized housing and other public benefits, and (if funding permits us to do follow-up surveys) health and well-being outcomes. With help from the Boston Housing Authority (BHA), Economic Mobility Pathways (EMPath) will recruit participants for the AMP Up Boston program from residents in BHA public housing and recipients of BHA housing vouchers and randomly assign access to the program among interested eligible residents. Treatment group participants will be able to receive three years of Mobility Mentoring Services, while control group participants will receive the services usually available to them in the community. We plan to follow study participants for ten years from random assignment through administrative data, allowing us to see whether recipients of Mobility Mentoring make temporary, short-term gains or sustainable, long-term changes in their economic situation.
External Link(s)

Registration Citation

Citation
Engle, Elizabeth, Lawrence Katz and Jonathan Tebes. 2021. "Supporting Pathways Out of Poverty: A Randomized Evaluation of AMP Up Boston." AEA RCT Registry. November 08. https://doi.org/10.1257/rct.8429-1.1
Experimental Details

Interventions

Intervention(s)
AMP Up Boston provides personal assistance through Mobility Mentoring, a system of goal setting, incentives, and individualized coaching designed to holistically address the problems participants face. Through regular meetings, mentors assess and track progress along five domains – family life, health, finances, education, and career. Mentors work with families to develop a customized plan to achieve economic independence, broken down into small, manageable steps. This involves setting goals to pay off debt, obtain additional schooling, apply for jobs, access healthcare, and obtain stable housing. Additionally, mentors offer targeted financial incentives to help encourage individuals to save, obtain training towards a new career, or take other steps toward the goals they have set for themselves. The AMP Up Boston program lasts three years.
Intervention Start Date
2021-10-27
Intervention End Date
2025-10-27

Primary Outcomes

Primary Outcomes (end points)
Our primary outcome is annual labor earnings (or quarterly earnings, if available).
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Our secondary outcomes are:
- employment (an indicator for whether an individual is employed)
- high-wage employment (an indicator for whether an individual is employed in a high-wage job)
- total annual household income
- housing stability and housing support (receipt of rent subsidy, amount of rent subsidy, homelessness, neighborhood poverty rate)
- personal finance (credit score, overdue debts, delinquencies) from credit bureau data
- education and training (completion of high school diploma/GED/HiSET, completion of certification, completion of associate's degree, completion of bachelor's degree, training and employment program participation)
- health and well-begin (survey measures of self-assessed physical and mental health if follow-up survey funding available)
- public assistance (receipt and amount of SNAP and TANF benefits)

Subgroup Analyses
We plan to examine heterogeneity of effects on the listed primary and secondary outcomes by:
- race
- primary language
- employment status at start of study
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
The study will recruit participants for the AMP Up Boston program from among eligible residents in BHA public housing and recipients of BHA housing vouchers. Since more residents are likely to be interested in Mobility Mentoring than EMPath can serve, we will randomize access to the program among interested eligible residents. This will create two similar groups, one which is offered Mobility Mentoring through AMP Up Boston (the treatment group) and one which is not (the control group). We can then compare outcomes between the two groups to see what difference Mobility Mentoring makes for those who receive the program, on top of the gains they would have made on their own as seen in the control group. AMP Up Boston participants will be able to receive three years of Mobility Mentoring Services. We plan to follow study participants’ outcomes for ten years from random assignment through administrative data, allowing us to see whether recipients of Mobility Mentoring make temporary, short-term gains or sustainable, long-term changes in their economic situation.
Experimental Design Details
Not available
Randomization Method
Computer random number generator
Randomization Unit
Individual
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
N/A
Sample size: planned number of observations
570. We will end recruitment once we have 200 active program participants in the treatment arm, so this number may be smaller or larger depending on the take-up rate of the program among the treatment group. Also, if funding can be attained to hire additional mentors and serve more participants, we will try to increase the sample size to 267 active program participants and increase the treatment and control groups (with a 50 percent treatment randomization rate) to reach this number receiving Mobility Mentoring Services.
Sample size (or number of clusters) by treatment arms
285 treatment participants, 285 control participants. We will end recruitment once we have 200 active program participants in the treatment arm, so this number may be smaller or larger depending on the take-up rate of the program among the treatment group. Also, if funding can be attained to hire additional mentors and serve more participants, we will try to increase the sample size to 267 active program participants and increase the treatment and control groups (with a 50 percent treatment randomization rate) to reach this number receiving Mobility Mentoring Services. Finally, if the take-up rate of services among the treatment group is below 70%, we may continue recruitment until we achieve 80% power to detect the minimum detectable effects noted in our power calculations.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Using historical data on employment and earnings for previous Mobility Mentoring enrollees, we estimate the minimum detectable effect (MDE) among the treated (i.e. treatment-on-treated (TOT)) necessary to observe a treatment effect with 5% statistical significance at 80% power. We assume a sample size of 580 study participants, 50% of whom are randomly assigned to the treatment group and offered the opportunity to enroll. We further assume an effective take-up rate of 70%, meaning that the difference between the treatment and control group participation rate is 70%. (e.g. 75% in the treatment group vs. 5% in the control group). Finally, we assume that controlling for the lagged dependent variable soaks up 20% of outcome variance. These calculations suggest that this study has sufficient power to detect a 20% or 14 percentage point increase in employment rates and a 34% or $4,180 increase in annual earnings. Provided additional funding, the expanded sample that would increase the sample size of active program participants to 267 has power to detect a 17% or 12 percentage point increase in employment rates and a 29% or $3,609 increase in annual earnings. These gains are within the range of observational changes in income and employment; 97% of graduates of EMPath’s flagship program were employed at exit relative to 65% at entry. Graduates also saw their annual incomes rise by 96% from $23,000 to $45,000 on average. These gains, however, may be an artifact of low-achieving participants disproportionately attritting from the sample or regression to the mean (if participants enroll during acute economic hardship). RCTs of intensive job training and employment programs, such as Job Corps, Project QUEST, Year Up, and the Sectoral Employment Impact Study (SEIS), provide information on possible treatment effect ranges as well as control-group trajectories absent treatment (Katz, Roth, Hendra, and Schaberg 2020). Treatment effects on earnings at several years out range between 8.1% (Job Corps) and 30 to 40% (Year UP) and employment effects range between 3.5% (Job Corps) and 20.3% (SEIS). Therefore, the present study is powered to detect treatment effects of effective employment programs, such as SEIS and Year Up. Given that Mobility Mentoring offers more comprehensive support services than job training and employment programs, we view the treatment effects of effective employment programs as a plausible lower bound on the range of policy-relevant treatment effects for Mobility Mentoring. The earnings gains observed among low-income single-parent households in the control groups of Project QUEST and SEIS studies were from 38 to 55%. This study therefore still has sufficient power to detect observed earnings gains even if a substantial portion of pre/post gains observed for past participants resulted from regression to the mean (i.e., 96%- 55% = 41%). References: Katz, Lawrence F., Jonathan Roth, Richard Hendra, and Kelsey Schaberg. 2020. “Why Do Sectoral Employment Programs Work? Evdence from WorkAdvance.” NBER WP No. 28248.
IRB

Institutional Review Boards (IRBs)

IRB Name
Harvard University-Area IRB
IRB Approval Date
2021-10-12
IRB Approval Number
IRB21-0949