Supporting Evidence Building in Child Welfare Project – LifeSet Evaluation

The LifeSet program is operated by Youth Villages, a private nonprofit organization. Formerly known as Transitional Living, LifeSet provides at-risk youth and young adults leaving foster care, juvenile justice, and mental health systems with the intensive in-home support and guidance they need to make a successful transition to adulthood. The core of the program is a therapeutic case management relationship between participant youth and “LifeSet specialists,” who work with youth to develop and move towards youth-defined goals in multiple domains of independent living including education, housing, employment and financial security, health and safety, and social connections and support. Youth Villages is currently operating in seven states in the US—Georgia, Massachusetts, Mississippi, North Carolina, Oklahoma, Oregon, and Tennessee—and the organization trains and supports nonprofit partners to offer LifeSet in other communities in six additional states. The CWEST project is evaluating LifeSet in New Jersey, which will be delivered by four local providers in the state.

When a young person enters the LifeSet program, the LifeSet specialist works with the youth to develop an individualized plan based on the youth’s personal context and goals. After the initial assessment, the youth and the LifeSet specialist meet at least weekly for one-hour, face-to-face or virtual sessions. During these sessions, the LifeSet specialist provides clinically focused case management, support, and counseling aimed at helping the youth move towards his or her goals. These meetings occur in a community setting, either at the youth’s home, work, or at whatever other location is most comfortable and convenient for him or her. In addition to these regular, scheduled meetings, LifeSet assures that youth are able to get in touch with a LifeSet specialist 24 hours a day, 7 days a week. The maximum duration youth can receive LifeSet services in New Jersey is 12 months.

LifeSet specialists work with between eight and ten youth at any one time and have either a master’s degree in a relevant field, or a bachelor’s degree and relevant work experience. Team supervisors supervise four LifeSet staff, including conducting group supervision, individual and field supervision, and professional development, and have a master’s degree in a relevant field, or a bachelor’s degree with relevant work experience. Licensed Program Experts are responsible for ensuring model fidelity and ideally have LifeSet implementation experience at the specialist and supervisor level and hold a master’s degree and professional license in a social service field. Licensed Program Experts meet regularly with each LifeSet team to provide clinical supervision and guidance on the program model.

2021-08-16

2024-08-31

The primary research questions for the impact study are does LifeSet improve outcomes of target population youth in the domains of:
a. Connections to education and employment
b. Social connections
c. Housing stability
d. Youth well-being (including resilience and social-emotional competence)

Except where noted, survey items used to assess outcomes are generally taken from existing surveys with similar populations including the Midwest Study of the Adult Functioning of Former Foster Youth (Courtney et al., 2011), California Youth Transitions to Adulthood Study (CalYOUTH, Courtney et al., 2020), and the National Longitudinal Survey of Youth (NLSY, Bureau of Labor Statistics, 2013).
a. Connections to education and employment. The outcome will be measured as whether youth in the treatment group had a higher likelihood of (1) receiving a high school diploma or GED certificate, if not already received at baseline; (2) enrollment in or completion of postsecondary education or training; and (3) of being employed within 1 year and 2 years of randomization. Education outcomes will be measured through participants’ responses to a series of survey items and through administrative data from the National Student Clearinghouse. Employment outcomes will be measured through participants’ responses to a series of survey items and through administrative data from state wage records.
b. Social connections. The outcome will be measured as whether youth in the treatment group had a higher sense of social connection and support. This outcome will be measured through participants’ responses to a series of survey items from the Social Support Network Questionnaire (SSNQ) (Gee & Rhodes, 2007) and through data collected by the local LifeSet Provider on youths’ social connections at program exit.
c. Housing stability. The outcome will be measured as whether youth in the treatment group had a lower likelihood of (1) being housing unstable; (2) staying in an emergency homeless shelter at any time point; and (3) being placed in a transitional living arrangement. This outcome will be measured through participants’ responses to a series of survey items and through administrative data from the state child welfare agency and local Homeless Management Information System (HMIS) provider.
d. Youth well-being. The outcome will be measured as whether youth in the treatment group had higher-level feelings of resiliency and social-emotional competence. This outcome will be measured through participants’ responses to a series of survey items from the Youth Thrive survey (Center for the Study of Social Policy, 2020).

The secondary research questions for the impact study are does LifeSet improve outcomes of target population youth in the domains of:
a. Mental health
b. Contact with the criminal justice system
c. Intimate partner violence
d. Economic well-being

a. Mental health. The outcome will be measured as whether youth in the treatment group experienced with less frequency (1) symptoms of depression, anxiety, and stress; (2) symptoms of Post-Traumatic Stress Disorder (PTSD); and (3) psychiatric hospitalization. This outcome will be measured through participants’ responses to a series of survey items, including items from the Depression Anxiety Stress Scale-21 (DASS21) (Lovibond & Lovibond, 1995) and Primary Care PTSD Screen for DSM-5 (PC-PTSD-5) (Prins et al., 2015).
b. Contact with the criminal justice system. The outcome will be measured as whether youth in the treatment group were less likely to (1) be arrested; (2) be convicted of a crime; (3) spend at least one night in a correctional facility; and (4) be sentenced to probation or supervision. This outcome will be measured through participants’ responses to a series of survey items and through administrative data from state criminal history and court records.
c. Intimate partner violence. The outcome will be measured as whether youth in the treatment group were less likely to be victims or perpetrators of intimate partner violence. This outcome will be measured through participants’ responses to survey items from the Conflict Tactic Scale (CTS) (Straus et al., 1996).
d. Economic well-being. The outcome will be measured as whether youth in the treatment group were (1) less likely to experience economic hardships and food insecurity; (2) less likely to receive public benefits; (3) more likely to have a checking or savings account; and (4) more likely to have a higher amount of money in their checking or savings account. This outcome will be measured through participants’ responses to a series of survey items, including items from the USDA Food Security Survey (Carlson, Andrews, & Bickel, 1999) and administrative data from the state public benefits agency.

The LifeSet evaluation will employ experimental methods, i.e. a randomized controlled trial (RCT). The eligible population for LifeSet will include youth 17-21 who have an open DCF case, are in the custody or guardianship of DCF, are living in the community (i.e. not incarcerated, in residential treatment, or hospitalized), and do not have a disqualifying condition (e.g., serious violent criminal history, residence out of state, severe mental illness, or intellectual disability). The four local LifeSet providers serve the Southern, Northern, and Central Regions, covering 18 of New Jersey’s 21 counties.

Identification and randomization of target-age youth will be done by New Jersey DCF based on administrative records and information gathered from youths’ case workers. This process will be conducted by age cohort, starting with youth who are at least 19 years-old and moving to younger ages in successive monthly randomization rounds. Youth will be randomly assigned to LifeSet or services as usual prior to obtaining consent/assent for youths’ participation in evaluation data collection. New Jersey DCF will then refer youth in the treatment group to a LifeSet provider; youth in the control group will be referred to, or continue receiving, services as usual.

Randomization began in August 2021 using a 1:1 randomization ratio (1 treatment to 1 control, or 1T:1C). However, only 60 percent of treatment group youth had gone on to enroll in LifeSet by, much lower than the anticipated rate of 90 percent. To account for the decrease in excess demand, the randomization ratio was changed in March 2022 to a 2:1 randomization ratio (2 treatment to 1 control, or a 2T:1C). The anticipated sample size was increased from 600 to 661 to allow for as many eligible youth to be randomized into the treatment group as possible while preserving statistical power with unequally sized groups. A total of 340 youth were randomized at the 1T:1C ratio (170 in each group) and an expected 321 additional youth will be randomized at the 2T:1C ratio, for a final sample size of 661 (384 treatment, 277 control).

Not available

Identification of target-age youth will be done by New Jersey DCF based on administrative records and information gathered from youths’ case workers. New Jersey DCF will use its administrative data to identify youth who meet the age criteria for LifeSet and are not excluded due to other criteria identifiable in DCF administrative data. This list will be sorted first by region and then in descending order of youths’ age in days. The DCF Research Coordinator will conduct a secondary screening by contacting the caseworkers for youth deemed eligible based on administrative records to obtain caseworker confirmation that the youth does not meet any exclusion criteria.

New Jersey DCF will send the list of eligible youth and the number of study slots needed per region to the Urban Institute project team who will perform the random assignment. New Jersey DCF will send this list via CSV file to the Urban Institute research team. Urban will conduct the random assignment in pairs per region using STATA. Urban will then send the list with group assignments added back to DCF.

We note that the target population is not evenly distributed across the four providers’ catchment areas. To prevent one provider from randomly having all youth in their catchment area assigned to only one group, randomization will be stratified by provider catchment area using a 2T:1C ratio within each stratum.

Individual

No

NA

The study anticipates recruiting a total of 661 youth

The study estimates including 384 youth in the treatment group and 277 youth in the control group.

For the power calculation, we assume 384 participants in the treatment group and 277 participants in the control group. An R2 of 0.30 is assumed, as a conservative estimate based on a prior evaluation of LifeSet. We also assume that administrative data will be available for the full sample and that the survey response rate will be 85 percent. Minimum Detectable Effects (MDE)s were calculated for an alpha level of 0.05, two-tailed test with 80 percent power. For outcomes based on administrative data, we estimated an MDE of 0.185. For outcomes based on survey data, we estimated an MDE of 0.201, given estimated attrition.

LifeSet Analysis Plan