Factors affecting university application decisions

The intervention will involve providing secondary school students in the UK with information, mentorship, and subsidized in-person visits to universities, with the treatment arms varying the content and nature of each.

2024-10-01

2026-01-01

Short-run outcomes (collected at midline and endline surveys):
Beliefs about outcomes at universities where students are assigned videos, mentors, and visits
“How likely are you to make friends and fit in at [university]?”
“How likely are you to receive an offer from [university]”?
“How likely are you to graduate successfully from [university]?”
Stated preferences over universities and application intent

Long-run outcomes (collected by schools after university applications are submitted):
- Applications / attendance to universities where students are assigned videos*
- Applications / attendance to mentor's university
- Applications / attendance to university with subsidized visit

Note that we notionally assign each student in the control group in one of the treatment arms, and apply the same algorithm to assign them universities for videos, mentors and visits, but do not actually offer them these treatments. Having assigned universities for the control group allows the university- specific outcomes described above to be defined in the control group consistently, meaning that we can construct placebo versions of these outcomes in the control group for use in analysis. .

We will look into heterogeneity of treatment effects on the primary outcomes by demographic similarity between participating students and mentors (where demographics include socioeconomic status, ethnicity, gender, and location). For instance, we will examine whether female students matched with female mentors were more likely to apply to their mentor's university compared to female students matched to male mentors.

The experimental design consists of two cross-randomized treatments and a control group with randomization at the individual student level. All students will complete baseline, midline, and endline surveys before and after the intervention.

Not available

Randomization will be conducted on a computer. We randomize students between arm C and the treatment arms when they complete the baseline survey using randomization in Qualtrics, stratifying by school. As noted above, for Queen Elizabeth’s Grammar School Faversham, randomization in the baseline survey only randomized between arm C and a treatment arm providing both mentors and visits without distinctions between demographically matched and unmatched mentors.

Individual, stratified by school

No

As the randomization is at the individual level, the number of clusters at the unit of randomization level is the same as the number of individuals. However, we will be sampling particular schools, and our central estimate is that we will recruit 20 schools.

Our central estimate is that we will have a sample size of 2000 individuals, but we may get fewer or more as we are still in the process of recruiting schools to the study and confirming participation.

Subject to budgetary and logistical constraints, we plan to assign 50% of participating students to the active control group, and evenly assign students to each of the four treatment arms, so that 12.5% of students are in each of arms T1a, T1b, T2a and T2b. Given our central estimate of 2000 participating students, this implies that we will have 250 students in each treatment arm and 1000 in the control arm. If we encounter constraints on our budget or the capacity of our mentors, we will reduce assignment to treatment evenly across the four treatment arms.

Students in the control group C will be notionally assigned (for the purpose of constructing placebo outcomes) to one of the four treatment groups in even proportions; under our main plan this will mean notionally assigning them to each of the four groups with 12.5% probability.

We conduct power calculations via simulation for our main binary outcome of applications to university on the basis of the following assumptions: - The outcome has a baseline probability of 20% (this seems like a reasonable estimate for the specific universities that students are assigned) - We have 2000 participating students and assign each treatment with probability 12.5% at each school - We allow for positive spillovers between a students’ three best friends that they report in a school in the data-generation process - We regress the outcome on dummies for the treatment (or relevant pooled treatments) and a linear control for the number of friends treated (from 0-3) Under these assumptions, MDES are as follows for the following comparisons of interest, including some pooled comparisons: - T** vs. C (treatment effect of any given treatment arm): 8.5pp - [T2a + T2b] vs. [T1a + T1b] (marginal effect of demographic match): 8.5pp - [T1b + T2b] vs. [T1a + T2a] (marginal effect of visits): 8.5pp - [T1a + T1b + T2a + T2b] vs. C (pooled effect of any treatment): 5.1pp - [T2a + T2b] vs. C (effect of visits relative to control): 7.0pp