| Field | Before | After |
|---|---|---|
| Field Last Published | Before August 02, 2021 02:00 PM | After February 28, 2023 03:20 PM |
| Field Experimental Design (Public) | Before The LifeSet evaluation will employ experimental methods, i.e. a randomized controlled trial (RCT). The eligible population for LifeSet will include youth 17-21 who have an open DCF case, are in the custody or guardianship of DCF, are living in the community (i.e. not incarcerated, in residential treatment, or hospitalized), and do not have a disqualifying condition (e.g., serious violent criminal history, residence out of state, severe mental illness, or intellectual disability). The four local LifeSet providers serve the Southern, Northern, and Central Regions, covering 18 of New Jersey’s 21 counties. We estimate that the LifeSet program will be able to serve approximately 300 study-enrolled youth during the two-year intake period. Given a 1:1 ratio of assignment to treatment and control group, the study anticipates recruiting a total of 600 youth. Identification and randomization of target-age youth will be done by New Jersey DCF based on administrative records and information gathered from youths’ case workers. This process will be conducted by age cohort, starting with youth who are at least 19 years-old and moving to younger ages in successive monthly randomization rounds. Youth will be randomly assigned to LifeSet or services as usual prior to obtaining consent/assent for youths’ participation in evaluation data collection. New Jersey DCF will then refer youth in the treatment group to a LifeSet provider; youth in the control group will be referred to, or continue receiving, services as usual. | After The LifeSet evaluation will employ experimental methods, i.e. a randomized controlled trial (RCT). The eligible population for LifeSet will include youth 17-21 who have an open DCF case, are in the custody or guardianship of DCF, are living in the community (i.e. not incarcerated, in residential treatment, or hospitalized), and do not have a disqualifying condition (e.g., serious violent criminal history, residence out of state, severe mental illness, or intellectual disability). The four local LifeSet providers serve the Southern, Northern, and Central Regions, covering 18 of New Jersey’s 21 counties. Identification and randomization of target-age youth will be done by New Jersey DCF based on administrative records and information gathered from youths’ case workers. This process will be conducted by age cohort, starting with youth who are at least 19 years-old and moving to younger ages in successive monthly randomization rounds. Youth will be randomly assigned to LifeSet or services as usual prior to obtaining consent/assent for youths’ participation in evaluation data collection. New Jersey DCF will then refer youth in the treatment group to a LifeSet provider; youth in the control group will be referred to, or continue receiving, services as usual. Randomization began in August 2021 using a 1:1 randomization ratio (1 treatment to 1 control, or 1T:1C). However, only 60 percent of treatment group youth had gone on to enroll in LifeSet by, much lower than the anticipated rate of 90 percent. To account for the decrease in excess demand, the randomization ratio was changed in March 2022 to a 2:1 randomization ratio (2 treatment to 1 control, or a 2T:1C). The anticipated sample size was increased from 600 to 661 to allow for as many eligible youth to be randomized into the treatment group as possible while preserving statistical power with unequally sized groups. A total of 340 youth were randomized at the 1T:1C ratio (170 in each group) and an expected 321 additional youth will be randomized at the 2T:1C ratio, for a final sample size of 661 (384 treatment, 277 control). |
| Field Randomization Method | Before Identification of target-age youth will be done by New Jersey DCF based on administrative records and information gathered from youths’ case workers. New Jersey DCF will use its administrative data to identify youth who meet the age criteria for LifeSet and are not excluded due to other criteria identifiable in DCF administrative data. This list will be sorted first by region and then in descending order of youths’ age in days. The DCF Research Coordinator will conduct a secondary screening by contacting the caseworkers for youth deemed eligible based on administrative records to obtain caseworker confirmation that the youth does not meet any exclusion criteria. New Jersey DCF will send the list of eligible youth and the number of study slots needed per region to the Urban Institute project team who will perform the random assignment. New Jersey DCF will send this list via CSV file to the Urban Institute research team. Urban will conduct the random assignment in pairs per region using STATA. Urban will then send the list with group assignments added back to DCF. | After Identification of target-age youth will be done by New Jersey DCF based on administrative records and information gathered from youths’ case workers. New Jersey DCF will use its administrative data to identify youth who meet the age criteria for LifeSet and are not excluded due to other criteria identifiable in DCF administrative data. This list will be sorted first by region and then in descending order of youths’ age in days. The DCF Research Coordinator will conduct a secondary screening by contacting the caseworkers for youth deemed eligible based on administrative records to obtain caseworker confirmation that the youth does not meet any exclusion criteria. New Jersey DCF will send the list of eligible youth and the number of study slots needed per region to the Urban Institute project team who will perform the random assignment. New Jersey DCF will send this list via CSV file to the Urban Institute research team. Urban will conduct the random assignment in pairs per region using STATA. Urban will then send the list with group assignments added back to DCF. We note that the target population is not evenly distributed across the four providers’ catchment areas. To prevent one provider from randomly having all youth in their catchment area assigned to only one group, randomization will be stratified by provider catchment area using a 2T:1C ratio within each stratum. |
| Field Planned Number of Observations | Before The study anticipates recruiting a total of 600 youth | After The study anticipates recruiting a total of 661 youth |
| Field Sample size (or number of clusters) by treatment arms | Before The study estimates including 300 youth in the treatment group and 300 youth in the control group. | After The study estimates including 384 youth in the treatment group and 277 youth in the control group. |
| Field Power calculation: Minimum Detectable Effect Size for Main Outcomes | Before For the power calculation, we assume 300 participants in the treatment group and 300 participants in the control group. An R2 of 0.30 is assumed, as a conservative estimate based on a prior evaluation of LifeSet. We also assume that administrative data will be available for the full sample and that the survey response rate will be 85 percent. Minimum Detectable Effects (MDE)s were calculated for an alpha level of 0.10 or 0.05, two-tailed test with 80 percent power. For outcomes based on administrative data, we estimated MDEs of 0.170 and 0.192 for alpha levels of 0.10 or 0.05, respectively. For outcomes based on survey data, we estimated MDEs of 0.185 and 0.208, given estimated attrition. | After For the power calculation, we assume 384 participants in the treatment group and 277 participants in the control group. An R2 of 0.30 is assumed, as a conservative estimate based on a prior evaluation of LifeSet. We also assume that administrative data will be available for the full sample and that the survey response rate will be 85 percent. Minimum Detectable Effects (MDE)s were calculated for an alpha level of 0.05, two-tailed test with 80 percent power. For outcomes based on administrative data, we estimated an MDE of 0.185. For outcomes based on survey data, we estimated an MDE of 0.201, given estimated attrition. |
| Field Public analysis plan | Before No | After Yes |
| Field Building on Existing Work | Before | After No |
| Field | Before | After |
|---|---|---|
| Field Document | Before |
After
LifeSet Analysis Plan.pdf
MD5:
ea5bc8c2158361c438c0d44749b97d54
SHA1:
3d271c5f5a2955445c4e0cc760e209e1c550f1d8
|
| Field Title | Before | After LifeSet Analysis Plan |