Impact Evaluation of Online Program Delivery of Catholic Charities Fort Worth’s LIFT Program

The LIFT program combines individualized case management, resource connections, financial coaching, and strategic financial assistance with the goal of advancing an individual's long-term financial freedom, labor market success, and well-being. In one-on-one consultations, paired navigators focus on three areas contributing to self-sufficiency: financial resilience, resource stability and emotional resilience, the ability to plan and cope. A tenet of the LIFT intervention is that a client’s readiness for change is essential to her or his success, and to achieve financial independence, individuals must actively contribute to creating their goals, objectives, and service plans.
Since LIFT’s inception in 2016, the program has helped more than 500 clients in the Dallas-Fort Worth area. An earlier RCT conducted collaboratively by CCFW and LEO evaluated a similar case management program by CCFW called Padua. The results of the study show that Padua led to a significant improvement in labor market and housing stability for those in the Padua program. CCFW’s strong culture of evaluation extends into their Research & Analytics Department, which tracks program data suggesting that LIFT similarly increases earnings, employment, and overall well being. Combining the evidence from the Padua RCT and LIFT program data, it is likely that LIFT is an effective program, but it is vital to evaluate the effectiveness of various delivery methods (in-person and virtual) before expanding the program to the more rural areas of CCFW’s 28 counties. Therefore, all study participants will take part in the LIFT program and have access to the same resources, but randomization will determine if their case management sessions with their navigator occur in person or online.

2025-02-04

2029-02-04

Program engagement, goal attainment index, credit score, employment

- Program engagement: Continued program engagement or successful program completion at 6, 12, and 18 months following randomization
- Goal attainment index: Rather than analyze multiple subgroups and outcomes in separate regressions, we will construct an index of goal attainment based upon study participants’ primary goal as stated at baseline. The index will be measured at 6, 12, and 18 months following randomization. The index will combine standardized measures of goal attainment with z-scores using the measure’s observed value for a given participant, the mean of the measure across the sample, and the standard deviation of the measure across the sample. To ensure comparability across different outcomes, we will recalibrate the direction of z-scores so that positive values consistently represent progress toward goals (e.g., a decrease in debt will be transformed to result in a positive z-score). The index will include the following measures, where the stated goal is listed and followed by the measure of attainment:
1. Increasing savings: standardized measure for increase in self-reported savings
2. Decreasing debt: standardized measure for decrease in Experian-reported total balance on delinquent accounts
3. Building credit: standardized measure for increase in Experian-reported credit score
4. Housing stability: standardized measure for count of Infutor-reported address changes
5. Career/job opportunities: standardized measure for measure of UI earnings
6. Education/certification: standardized measure for NSC-reported initiated enrollment or degree/certificate completion
7. Emotional health/stress reduction: standardized measure for increase in self-reported emotional resilience
We will also report average treatment effects for each component of this index separately as secondary outcomes.
- Credit score: constructed from Experian data measured quarterly
- Employment: constructed from state administrative data-reported UI benefits

Program engagement, debt, degree enrollment/completion, housing stability, benefits usage, self-reported trust in navigator, resource stability, emotional resilience, financial knowledge and stability

Self-reported trust in navigator, resource stability, emotional resilience, and financial knowledge/stability outcomes will draw upon clients’ responses to three surveys CCFW administers at 6, 12, and 18 months during LIFT program engagement. Further details on secondary outcomes can be found in the attached analysis plan.

The CCFW team and members of LEO's research team collaboratively designed an RCT to evaluate the impact of virtual program delivery on the outcomes of clients in the LIFT program. The following elements describe the experimental design, beginning with study enrollment:
Starting in February 2025, individuals will be referred or contact CCFW enquiring about services. They will be directed to complete an online application for the LIFT program. Based upon their online application responses, clients will be evaluated for their study eligibility. Eligible applicants to the LIFT program will undergo a study consent process during their program intake. Eligible clients who provide study consent will be randomized during an intake call with an Operation Specialist. Then, a Program Manager will assign them to a virtual or in-person navigator based upon their randomized group assignment. If an individual does not consent to be part of the study, the outreach specialist will refer him or her to Padua, CCFW's holistic case management services. Navigators will call randomized clients with an initial welcome call within 2 days of assignment to administer baseline surveys and to schedule the first assessment visit. This first assessment visit will initiate randomized activities as the virtual group will continue program delivery via calls while the in-person group will continue program delivery in-person with their navigator. The content and resources associated with the LIFT program itself will be consistent across the randomized groups, and only the delivery method differs.
As part of internal program evaluation for clients in LIFT, CCFW invites their clients to complete a set of surveys covering topics of emotional resilience, resource stability, trust in navigator, and financial knowledge. These surveys are collected as part of regular contact between navigators and clients, and these responses will be used in the construction of study outcomes. Longitudinal data will be accessed using a partnership The Ray Marshall Center at the University of Texas. This will include linking to data on earnings, employment, public benefits use, as well as other records. Credit reports and consumer reference data for study participants will be accessed via LEO’s existing partnership with Experian, as well as residential address data via a similar partnership with Infutor.

To verify that the outcomes of LIFT participants are similar to those of the evidence-based Padua program, applicants to CCFW’s Padua will undergo a consent process during the study’s time period in which willing participants will provide permission for release of identifiable data to researchers. This will merely add a descriptive element to the study and allow some comparison to aggregate outcomes of clients across both programs.
The study will enroll approximately 1,300 participants over four years. Eligible clients include those who are 18 and above, speak English or Spanish, are ready and able to work, and live or work within the 28-county service area. Anyone with a financial coaching need is eligible; there are no income limitations on the sample.
By comparing the outcomes of individuals who were randomly selected to receive virtual case management (“Treatment”) against individuals who were randomly selected to receive in-person case management (“Control”), LEO will be able to isolate the causal effect of the virtual program on outcomes of interest. Program effects will be estimated by an intent-to-treat design where outcomes of interest will be regressed on a vector of observed characteristics, and a treatment group dummy variable. We will also report treatment-on-treated effects using a two-stage least squares regression approach. Heterogeneous impacts can be obtained by estimating the baseline regression with additional controls for groups and group-by-treatment interaction effects.
Following the planned four years of enrollment, two-year results for the full study sample will be available in 2030.

Not available

Computer (STATA)

Individual

No

1,300 individuals

1,300 individuals

Approximately 650 individuals will be enrolled in the treatment group (virtual services), and 650 will be enrolled in the control group (in-person services) over the course of the study.

We plan to enroll a study sample of 1,300 individuals, with approximately half of these assigned to the treatment group (virtual coaching), and a program take-up rate of 100%. We assume a high take-up rate due to the fact that randomization occurs after they have initiated program contact with CCFW staff but before any randomized activities begin. We are powered to detect a 6.7 percentage point change in the rate of continued engagement or program completion at 12 months following randomization. If we are unable to reject the null hypothesis that there is no significant difference between the outcomes of the randomized groups, we will be able to conclude that virtual program delivery has no impact, and is, in terms of outcomes, the same as in-person program delivery.

LIFT Pre-Analysis Plan