A Randomized Evaluation of STEM Focused Summer Programs

The evaluation examines three STEM-Focused summer programs: a six-week residential program, a one-week residential program, and an online program. The host institution office operates these three programs and recruits students between their junior and senior year of high school who face significant challenges to attending college or pursuing a STEM career. These challenges include:
• being the first in their family to attend university;
• an absence of science or engineering degrees in their family;
• attending a high school that sends fewer than 50 percent of its students to four-year colleges;
• being a member of an underrepresented minority group; and
• attending a high school that is rural or predominantly minority.

These programs vary in delivery methods and dosage, but all provide STEM-related coursework, college application and financial aid information, access to STEM mentors, and connections to high-achieving peers from underrepresented backgrounds with STEM interests. The six-week program has been operating for over 40 years and brings about 80 students to the host institution’s campus for a residential program. The one-week program and online program began operation in 2011 in response to the high demand for the six-week program. The one-week program brings about 100 students to the host institution’s campus for a week residential program. The online program serves about 140 students for six-months and a five-day conference on the host institution’s campus.

2014-05-15

2017-01-15

Outcome data will come from surveys and administrative data from the host institution, the National Student Clearinghouse (NSC), and administrative records from 36 selective private universities.

The host institution office has and will continue to deliver endline surveys to experimental subjects once a semester after the summer programs conclude for that year. These surveys ask students about their academic, career, and self-confidence outcomes. These outcomes include planned college applications, college admissions, knowledge of colleges, knowledge of financial aid, interest in STEM, self-reports of self-efficacy, life, and study skills, planned or current enrollment, planned or current majors, and academic performance.
The host institution office has and will continue to collect administrative data from the host institution, NSC, and administrative records from 36 selective private universities to document student academic outcomes. These outcomes include college application, college admission, college enrollment, persistence, graduation, GPA, and college major.


The host institution office has and will continue to deliver endline surveys to experimental subjects once a semester after the summer programs conclude for that year. These surveys ask students about their academic, career, and self-confidence outcomes. These outcomes include planned college applications, college admissions, knowledge of colleges, knowledge of financial aid, interest in STEM, self-reports of self-efficacy, life, and study skills, planned or current enrollment, planned or current majors, and academic performance.
The host institution office has and will continue to collect administrative data from the host institution, NSC, and COFHE to document student academic outcomes. These outcomes include college application, college admission, college enrollment, persistence, graduation, GPA, and college major.

Between surveys and administrative data, there are a large number of outcome variables collected in this analysis. If these were analyzed individually, some would be expected to show significant differences between the programs, solely due to chance. To avoid emphasizing spurious results, outcomes are grouped into related families. Following Anderson (2008), each family is converted into an index according to the following procedure:
• For each individual outcome in the family, define the variable such that higher values are “better."
• Normalize each outcome into a z-score relative to the control group for that cohort. That is, subtract the cohort-specific control group mean and divide by the standard deviation
• Construct the weighted average of all the outcomes in the family. The weight on each outcome is the inverse of the covariance matrix of the outcomes. In the case that a component of the index is missing, the index is calculated as the weighted average of the remaining components
• Program differences are examined using indices as the main outcome measures as opposed to the component variables.
There are indices related to college preferences, knowledge about the college and financial aid application process, confidence, skills, and application strategy. We also construct indices to measure the selectivity of schools the student is admitted to, and the school a student matriculates in. Standalone outcomes are used for application, admission, and enrollment to elite technical schools and the host institution. We will use this outcome data to compare academic and career performance between the experimental groups.

We randomized into four groups of students: three intervention summer programs and one control group. The three programs in the evaluation are a six-week residential program, a one-week residential program, and an online program. The host institution operates these three programs and recruits students between their junior and senior year of high school from all across the United States who face significant challenges to attending college or pursuing a STEM career.

The control group was created through the random assignment process. The host institution office receives many more applicants than can be served by the programs. Following the host institution office’s normal procedures, 700 students of an applicant pool of about 2,000 were selected each year for the randomization process. The selection committee was made up of stakeholders, community members, and affiliates with longstanding ties to the outreach office. The selection committee ranked applicants in terms of suitability for the six-week program. The outreach office did the final program assignment by lottery after all students were ranked. Selection committee rankings were a holistic reflection of the applicant’s fit for the six-week program. This was assessed based on academic ability and interest in STEM, as indicated through grades, test scores, letters of recommendation, and application essays. In addition, the mission of the program to help traditionally underrepresented populations makes it appropriate to consider the following risk factors during the selection process on a holistic basis though no element in isolation guarantees admissions.

1. the individual would have been the first in the family to attend college;
2. there was an absence in the individual’s family of science and engineering backgrounds;
3. the individual’s high school historically sent less than 50% of its graduates to 4-year-colleges;
4. the applicant attended a school that presented challenges for success at an urban elite university (e.g., rural or predominantly minority);
5. the individual was a member of a group that is under-represented in the study of science and engineering (African American, Latino or Native American).

In addition to each student’s individual characteristics, rankings in 2015 and 2016 were also affected by regional weights designed to increase representation from less dense parts of the country.

In 2014, 653 applicants were ranked by the selection committee. 2014 acted as a pilot year in the sense that the host institution was exploring how the changes in the selection process would affect operations. As a result, nineteen students were admitted to the six-week program through the traditional process. These students are excluded from the experiment, and are labeled as “Certainty Spots.” The 634 students who were not offered a certainty spot were sorted by their rank and placed into one of three blocks based on two cutoffs. Students above the first cutoff were randomly offered admission to either the six-week or the one-week program. Students between the first and second cutoff were offered admission to the one-week or the online program. Finally, applicants below the second cutoff were offered admission to either the online program or no program (control group). The ranking cutoffs for being assigned to block one, two, or three were chosen based on capacity constraints for each program and maintained similar sample sizes for each treatment arm within a block.

In 2015, there were 701 ranked applicants. The randomization design differed in two ways from the previous year: there were only four “certainty spots.” In addition, the number of blocks was reduced to two. Applicants above a single ranking cutoff were randomly assigned to either the six-week, the one-week, or the online program. Applicants below the cutoff were assigned to either the online program or the control group, as in 2014. The reduction in blocks from three to two ensures that less extrapolation is necessary when comparing programs across blocks.

In 2016, there were 749 ranked applicants, and 1 applicant who withdrew his or her application during the randomization process (who we consider to have rejected the program offer). Unlike previous years, there were no “certainty spots” for 2016. For all three cohorts, the program assignment used gender as a randomization. Therefore, program spots were always divided evenly between female and male applicants in each block.

For a comparison of programs to the control group, full randomization would have been ideal, but the modified block designs used in 2014, 2015, and 2016 took into account the outreach office’s programmatic considerations. The outreach office was concerned that the most qualified candidates might have received no program and that relatively less qualified candidates might have received more intensive interventions.

The randomization process is visually represented in Figure 1 in the “supporting documents/materials” section of this registration.

Individual

No

No clusters.

We will analyze the outcomes of about 2,100 students that applied to the program over the course of 3 years.

six-week program = 250 students, one-week program = 308 students, online program = 472 students, and control = 1,073 students

In order to conduct the analysis described above, our sample must be of sufficient size to detect treatment effects. We used findings from Robles’ retrospective study of the six-week residential program as plausible estimates of treatment effect sizes, assuming two cohorts (2014 and 2015). We added a third cohort of randomization ex post. While we had enough power to detect effects in our initial power calculation for cohort 2014 and 2015, the additional year of randomization will increase our precision. Using estimates from the observational study, Robles found a 13.9 percentage point increase in matriculation at the host institution. The control group mean for matriculation at the host institution was 15.4%, with a standard deviation of 36%. With 122 students in the treatment and control groups (61 in each year) and a standard alpha level of 0.05, using a two-tailed test, we have a power level of 86%. Since the standard power level to minimize type II errors is 80%, we are powered to detect effect sizes similar to those found in the observational study. If we use a one-sided test (which assumes the effect of attending the six-week program can only be positive), our power level reaches 92%. This is a plausible assumption given the findings from the retrospective evaluation. With the additional cohort of randomization from 2016, our power will increase further. One concern is that our effect sizes will be smaller, since we will be comparing the six-week program to the one-week program. We are confident that we will still be able to detect effects for two reasons: first, we exceed the 80% threshold so we have some room to decrease power, and second, we will control for block when estimating treatment effects, which will account for some of the variance in outcomes and increase our power.