College Forward RCT Follow-Up: Labor Market Impacts

The intervention is described in detail in Castleman, Deustchlander, and Lohner (2024). A public version of the working paper is available here: https://edworkingpapers.com/sites/default/files/College_Forward_Paper_June24.pdf

2016-01-01

2033-01-01

The primary hypothesis we propose to test in this study is whether College Possible Texas advising increases participants’ annual earnings in at least the tenth year following expected high school graduation (depending on experimental cohort). Another primary hypothesis we propose to test is whether College Possible Texas advising increases the share of participants who are employed in at least three out of four quarters during the tenth year following expected high school graduation.

We will measure these employment and earnings outcomes by leveraging state employment data from the Texas ERC. The Texas ERC maintains individual-level longitudinal data linking students from the Texas K-12 system (Texas Education Agency data, or TEA) to both the Texas higher education data (Texas Higher Education Coordinating Board, or THECB) and the Texas workforce data (Texas Workforce Commission, or TWC). We will construct our measures of annual earnings for each focal year (e.g. the tenth year after expected high school graduation) by summing quarterly unconditional earnings data from the Texas Workforce Commission in that year. For instance, for the cohort graduating high school in 2017Q2, we would calculate a student’s annual earnings ten years after high school by summing quarterly earnings in 2027Q3, 2027Q4, 2028Q1 and 2028Q2; if a student had no observed employment during a particular quarter, then we would set their earnings to $0 and include this zero in the average. We will define a separate outcome for employment as equal to one if the student was employed during at least three out of four of the quarters of interest. We will obtain employment data through 2030Q2, such that we can observe at least ten years after expected high school graduation for our full sample, which consists of the high school graduating cohorts from 2017-2020.

Our secondary hypotheses are whether College Possible Texas increases
annual earnings and employment over more proximal time horizons: seven and nine years after
expected high school graduation.

Sample selection and randomization for the study has already been completed. The application pool contained students at 11 local high schools in Austin and Houston, Texas; we used high school as a randomization block. Each year, we assigned approximately 60% of the applicants from each high school who met the College Forward eligibility requirements to the offer of College Forward advising, an offer which virtually all students took up, or to a control group that received no advising services from College Forward. As we show in Table 2 of Castleman, Deustchlander, and Lohner (2024), the experimental sample is well-balanced on baseline characteristics. Our experimental sample size for the proposed study is the same as the sample size in the original RCT: 1,605 individuals across four cohorts (high school classes of 2017 - 2020), with 963 assigned to treatment and 642 assigned to control.

The experimental sample includes students who were offered College Possible Texas advising services. While we do not have data on actual take-up of the advising offer, College Possible Texas has shared with us that the vast majority of students who receive an offer join the program. Still, we will view our estimates as intent-to-treat (ITT). We will use a standard OLS regression model to estimate the ITT impact on unconditional earnings:

𝐸𝑎𝑟𝑛𝑖𝑛𝑔𝑠𝑖 = β0 + β1𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝐺𝑟𝑜𝑢𝑝𝑖 + β2𝑋𝑖 + 𝐹𝐸𝐻𝑆 * 𝐶𝑜ℎ𝑜𝑟𝑡+ ϵ𝑖

Where 𝑋𝑖 is a vector of available student baseline characteristics (cohort, gender, first generation status, free or reduced price lunch status, race and ethnicity, whether English is spoken at home),and 𝐹𝐸 is a set of high school by cohort fixed effects. We include the latter because the 𝐻𝑆 * 𝐶𝑜ℎ𝑜𝑟𝑡 student-level randomization was stratified within high school and cohort. The coefficient of interest is β1, which provides ITT estimates of assignment to College Possible Texas advising. We will repeat the same OLS regression model using a binary indicator for employment as the
outcome.

Following Castleman, Deutschlander, and Lohner (2024), to account for modest crossover between experimental groups we will also estimate an instrumental variables (IV) model in which we will use students’ original experimental group assignment as an instrument for the
offer of College Forward advising; this will yield a crossover-adjusted ITT estimate.*** The IV model takes the following form:

𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑂𝑓𝑓𝑒𝑟𝑖 = α0 + α1𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝐴𝑠𝑠𝑖𝑔𝑛𝑚𝑒𝑛𝑡𝑖 + α2𝑋𝑖 + 𝐹𝐸𝐻𝑆 * 𝐶𝑜ℎ𝑜𝑟𝑡+ 𝑢𝑖

𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖 = γ0 + γ1𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑂𝑓𝑓𝑒𝑟𝑖 + γ2𝑋𝑖 + 𝐹𝐸𝐻𝑆 * 𝐶𝑜ℎ𝑜𝑟𝑡 + ϵ𝑖

where 𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝐴𝑠𝑠𝑖𝑔𝑛𝑚𝑒𝑛𝑡𝑖 is the student’s original randomization group and 𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑂𝑓𝑓𝑒𝑟𝑖 corresponds to the student’s ultimate treatment status. These variables only differ for the 50 crossover students. In Castleman, Deustchlander, and Lohner (2024), we find that the main ITT and IV impacts on enrollment differ by only 0.5 percentage points, or 7 percent (Table 3). We will report both ITT and IV impacts on our primary and secondary labor market outcomes.

Given that all students in the experimental sample attended public Texas high schools, we expect a near 100% match rate within the Texas ERC database. However, if any individuals are working outside the state of Texas or are working in positions not covered by the TWC data (e.g. independent contractors), then we will not observe their employment and will be unable to distinguish these individuals from those who are truly unemployed. We will treat all individuals without employment and earnings records in the TWC data as unemployed ($0 earnings). We will examine the data for outliers, and bottom or top code the earnings data accordingly. For instance, if there are any quarterly earnings either so large (e.g. $300,000) or so small (e.g. $10) that we could reasonably expect these values to impact the OLS estimates, then we will top or bottom code earnings to the 99.5th percentile and 0.5th percentile, accordingly. We expect that the number of adjustments will be minimal, if any.

*** Specifically, 25 individuals assigned to the treatment group ended up in the control group, and vice versa.

Not available

The randomization design is described in detail in Castleman, Deustchlander, and Lohner (2024). A public version of the working paper is available here: https://edworkingpapers.com/sites/default/files/College_Forward_Paper_June24.pdf

The randomization design is described in detail in Castleman, Deustchlander, and Lohner (2024). A public version of the working paper is available here: https://edworkingpapers.com/sites/default/files/College_Forward_Paper_June24.pdf

Yes

44 (11 high schools x 4 cohorts)

1605

60% within randomization block assigned to treatment, for a total of 963 treated and 642 count students

For our statistical power calculation, we used a simulation method to calculate power for a variety of potential effect sizes. We use simulations because we do not observe the mean or standard deviation of wages for our control group and all other estimates we found in publicly-available data of the means and standard deviations for Texas workers from similar backgrounds are conditional on employment. Additionally, the distribution of unconditional wages is highly skewed, with a concentration of zero wages among those unemployed, which does not conform with the normal distribution assumption of most power calculators. Based on our simulation, we are well powered (power >= 0.8) for an effect size of approximately $3,000 or larger on unconditional annual earnings. We describe our simulation method in the Appendix. Based on the impact estimates of College Possible Texas on bachelor’s degree attainment from our working paper, we believe that $3,000 (or higher) in unconditional annual earnings is a reasonable anticipated effect size. For each student induced to earn a BA degree (6.5 percentage points of the treated group), we anticipate the average gain in unconditional earnings to be $36,718 (converted to 2022$), or an average treatment effect across the full treatment group of $36,718 * 6.5 pp = $2,387.*** However, $2,387 assumes that only 6.5 percent of treated students (i.e. those induced to earn a Bachelor’s degree due to the intervention) received any labor market benefit from the intervention. Due to the extensive programming offered via College Possible Texas, we expect that other students benefited meaningfully, though more modestly, from the intervention. Specifically, College Possible Texas provided ongoing advising support for students beginning in college and continuing until they earned a postsecondary credential, for a total time of four to six years. This advising included discussion of a variety of career options as well as development of career skills, including resume building, writing a cover letter, and interview preparation. Therefore, we expect both treated students who would have graduated absent the intervention and treated students who did not graduate despite the intervention to have some average increase in earnings. We assume this increase would be much smaller than the increase in earnings stemming from obtaining a bachelor’s degree. Using a conservative estimate that the effect of career advising and preparation on earnings is 2-4 percent of the effect of obtaining a degree would be approximately $1,100 per student. This would make the average treatment effect across the full treatment group equal to ($1,100 * 93.5pp) + $2,387 = $3415. *** This anticipated gain is made up of two components: (1) Among students who would have been employed absent the intervention, and for whom treatment would affect earnings conditional on their employment, we estimate an average increase in earnings of $32,770.We obtain this estimate from Ma & Pender (2023), who compare earnings of workers with a Bachelors degree to workers with a high school diploma. We use the high school diploma group as the comparison group because we also found that the intervention increased any college enrollment by 7.3 percentage points. (2) Among students who were induced to employment due to treatment, we estimate the treatment effect on wages of $45,929. We base this on the median of all Hispanic Texas workers age 24-30 with a Bachelor’s degree or higher (from the 2022 American Community Survey) compared to the $0 wages earned by unemployed individuals. Assuming a 70% employment rate absent intervention (again from the 2022 ACS), then the expected wage gain for a treated student induced to earn a Bachelor’s degree would be (32,770 * 70% employed absent intervention) + ($45,929 * 30% unemployed absent intervention) = $36,718. Appendix: Power Calculation Simulation Method We build a “null” distribution of wages using the distributional points from the 2022 ACS 5-year estimates, with the sample including Hispanic Texas workers aged 24-30 with a high school diploma. The mean nonzero wage is $33,285. This null distribution of wages also assumes that 30% of individuals are unemployed (again, based on the 2022 ACS), and have wages equal to zero. The mean unconditional wage of the null distribution is $22,971. We then build a variety of test distributions of wages, reflecting effects on both conditional wages (X% increase in wages among individuals with nonzero wages in the null distribution) and employment (Y percentage point increase in individuals moving from zero wages to nonzero wages). For individuals with zero wages in the null distribution and nonzero wages in a test distribution, we assume the nonzero wages are randomly and uniformly distributed between the 25th and 75th percentiles we use above. For a given test distribution, we repeat the following 3,000 times: (1) Randomly assign n = 642 individuals (the number of control students in the intervention) to the null distribution of wages, and the other n = 963 individuals to the test distribution of wages; then (2) Regress wages on the treatment indicator and a simulated variable that accounts for one-third of the variation in the null wage distribution. This simulated variable is analogous to the baseline covariates that we will be able to include in our regression models estimating the impact of the intervention on labor market outcomes, which will increase the precision of our estimates. The baseline covariates we will include are high school fixed effects, year cohort, gender, race/ethnicity, first generation status, free/reduced price lunch status, and whether English was spoken at home. We calculate the share of the 3,000 repetitions where the p-value for the treatment indicator is < 0.05. This is our estimate of power: the likelihood of uncovering a true treatment effect when one exists.

Pre-analysis plan