Unpacking the Long-Term Impact of Holistic Supports for Community College Students: Evidence from a Randomized Controlled Trial

OMD is a non-profit organization serving community college students in the Chicago metro area that supports students financially, academically, socially, and professionally.

2016-01-01

2021-06-30

The proposed follow up study will track associate degree completion, transfer to 4-year institutions, 4-year degree completion, and employment outcomes up to nine years after randomization.

The proposed follow up study will analyze the cost-effectiveness to understand the cost of the improved outcomes that result from program participation.

The original study was an RCT in which randomization was implemented at the student-level, blocking on the campus the applicant was enrolled in or intended to enroll in and whether the applicant was a graduating high school student or a current community college
student. The research team will use regression analysis using a linear probability model to estimate intent-to-treat effects with fixed effects at the randomization block level, and will use two-stage least squares regression with the treatment assignment as an instrument to
estimate the effects of treatment-on-the-treated.

Not available

Prospective and current community college students applied to the OMD program through the standard application process. OMD screened applicants for program eligibility and the complete list of eligible applicants was sent to the research team. The research team then implemented permutation randomization in STATA, blocking on intended campus and high school or current community college status.

Randomization was conducted at the student-level.

No

N/A

In spring 2016, a total of 1,452 unique, eligible OMD applicants were randomized, 779 of whom were offered a spot in the program. In spring 2017, 2,821 unique, eligible OMD applicants were randomized, 1,396 of whom were offered a spot in the program. In spring 2018, 623 unique, eligible OMD applicants were randomized, 398 of whom were offered a spot in the program. Across these three cohorts, a total of 4,896 unique applicants were randomized, 2,573 of whom were offered a spot in the program.

2,567 scholars were randomized into the treatment group. 2,311 scholars were randomized into the control group.

With an analytic sample of 4,878 individuals, we are confident that we are sufficiently powered to detect substantively meaningful effects on our primary outcome of interest. Given a two-tailed test, a significance level of 0.05 and .80 power and more than 100 degrees of freedom (number of students in the sample less the number of blocks less the number of covariates), Schochet (2008) gives a value of 2.80 for this constant. Based on these calculations we estimate that the study is powered to detect a minimum detectible effect size (MDES) of 0.08 standard deviations. For our primary outcomes of interest (transfer, four-year degree completion, and employment rate), we draw on the existing literature to estimate baseline outcomes in order to translate the MDES into a more meaningful metric. The CCC FY21 Statistical Digest estimated that the transfer-out rate for first-time full-time degree seeking students within three years of enrollment is 13 percent (CCC, 2022). Using this as our baseline transfer rate, we estimate that the proposed study is powered to detect a change of 2.7 percentage points. The Community College Research Center estimates Illinois’ community college cohort bachelor’s completion rate to be 17 percent (Jenkins & Fink, 2016). Using this as a baseline, we estimate the current study is powered to detect a change in bachelor’s degree completion of 3 percentage points. The Current PopulationSurvey estimates that the unemployment rate for individuals with some college and no degree was 8.3 percent and 7.1 percent for those with an associate degree in 2020 (CPS, 2020). Based on the associate degree completion for the study control group of 26 percent, we use a baseline unemployment rate of 7.4 percent to estimate that the proposed study will be powered to detect an effect of 2 percentage points.