Factors affecting university application decisions

The intervention will involve providing secondary school students in the UK with information, mentorship, and subsidized in-person visits to universities, with the treatment arms varying the content and nature of each.

Low-income students in the UK “undermatch”, attending lower-ranked universities despite good academic credentials and uniform tuition fees. This project looks into whether and how social exposure to individuals who attend high-ranked universities shapes university aspirations. Using administrative data, we document that school-to-university pipelines are persistent and dense; on average, 32% of students attend the modal university attended by students in the prior cohort at their school. These effects are more pronounced in economically disadvantaged schools, where students who attend pipeline universities are also more likely to undermatch. Additionally, students are more likely to attend a university their schoolmates have historically attended, even when they have offers from higher-ranked universities. In light of these patterns, we propose learning about university opportunities through schoolmates as one driver of undermatching. To obtain causal evidence and disentangle mechanisms of this type of social learning, we will run an RCT on ~2,000 students in the UK applying to university. Mechanisms we will test include the effectiveness of social exposure compared to standard information interventions and whether students respond more to information from individuals who are demographically similar.

The intervention will involve in-person workshops in secondary schools in the UK providing information about the university application process to participants, video clip anecdotes from current university students, matching participants with current university student mentors to discuss life at university and the application process, and subsidized in-person visits to universities. Our treatment arms will vary whether students see videos, are matched to mentors, and are subsidised for visits. Each treatment arm will then be cross-randomized for mentors who are demographically similar to the student (share at least one of the characteristics of gender, ethnicity, and region with the student) or mentors who are dissimilar (share none of these characteristics with the student). The target population is Year 12 and/or Year 13 secondary school students in the UK in the year before they apply to university (typically January of Year 13). We plan on conducting the intervention in multiple waves in order to maximize the number of schools that can be included in the study. The first wave will take place in October through November 2024, and the second wave will take place primarily between May through July 2025, with some earlier interventions (detailed in the “experimental details” section below).

2024-10-01

2026-01-01

Short-run outcomes (collected at midline and endline surveys):
Beliefs about outcomes at universities where students are assigned videos, mentors, and visits
“How likely are you to make friends and fit in at [university]?”
“How likely are you to receive an offer from [university]”?
“How likely are you to graduate successfully from [university]?”
Stated preferences over universities and application intent

Long-run outcomes (collected by schools after university applications are submitted):
- Applications / attendance to universities where students are assigned videos*
- Applications / attendance to mentor's university
- Applications / attendance to university with subsidized visit

Note that we notionally assign each student in the control group in one of the treatment arms, and apply the same algorithm to assign them universities for videos, mentors and visits, but do not actually offer them these treatments. Having assigned universities for the control group allows the university- specific outcomes described above to be defined in the control group consistently, meaning that we can construct placebo versions of these outcomes in the control group for use in analysis. .

We will look into heterogeneity of treatment effects on the primary outcomes by demographic similarity between participating students and mentors (where demographics include socioeconomic status, ethnicity, gender, and location). For instance, we will examine whether female students matched with female mentors were more likely to apply to their mentor's university compared to female students matched to male mentors.

The experimental design consists of two cross-randomized treatments and a control group with randomization at the individual student level. All students will complete baseline, midline, and endline surveys before and after the intervention.

The intervention will be randomized at the student level and consist of two cross-randomized pairs of treatment arms and an active control group. All participating students will attend in-person workshops led by a current university student or recent university graduate and fill out a baseline survey ahead of the workshops about their university aspirations, and a midline survey at the end of the workshop. The baseline surveys will elicit students’ grades, their intended 5 universities for applications, beliefs about different universities, what they believe is the most common university destination at their school; the midline surveys will repeat questions about beliefs and intended universities of application, and additionally elicit student demographics, whether their parents or siblings went to university and which university(s) they went to if so, and preferences for mentors and visits.

Based on their responses to the baseline survey, participating students will be assigned to an “academic cell”: we group universities into three tiers based on the 25th percentile of ‘UCAS tariff points’ (a metric that standardizes performance across different high-school qualifications in the UK) of university enrollees in that tier, and determine a student’s cell based on the highest tier of university for which their predicted UCAS tariff points exceed the 25th percentile cutoff for that tier. The midline survey will ask all students to state universities for which they would like mentors and where they would like a subsidized visit to, along with repeating belief questions. After students in the relevant treatment groups have conducted their mentorship calls and visits, all students will conduct an endline survey that elicits intended applications and the same beliefs elicited in the baseline and midline surveys to measure any belief updating.

Students in the active control group C will attend an in-person workshop that includes generic information and application tips such as information about the university application process in the UK, timelines, and personal statement advice, as well as national and school-specific statistics on applications and universities. Students in treatment arms will attend the same workshop and similarly take baseline, midline and endline surveys. During the baseline survey, they will be shown videos of students from the same academic cell discussing their experiences applying to and attending university. After the workshop and midline survey, students in the treatment arm will also be matched with mentors. In arm T1, the mentors that students receive will be drawn from their academic cell, but will not share any demographic characteristics (gender, ethnicity, and region of the UK) with the student: they will be from a different gender, a different ethnicity and a different region. In arm T2, the mentor will share at least one of these characteristics with their mentee. We will then cross-randomize offering students in both of these arms a subsidy of up to £75 to visit a university of their choice: students in T1a and T2a will receive mentors but no visits, while students in T1b and T2b will receive both mentors and subsidized visits.

Two schools - Queen Elizabeth’s Grammar School Faversham and Harris Academy Bermondsey - required workshops to take place earlier in the application cycle than the planned intervention start date of May 2025. Because of this, some of the experimental details for these studies are different as they take place while we were finalising the design for our second wave based on feedback from grant reviewers. Students from Queen Elizabeth’s were assigned either to an arm where they received both mentors and visits, or an arm where they received neither, and students were not assigned into a treatment arm with demographically matched or unmatched mentors - instead, assignment of mentors happened without regard to demographic match, although the assignment procedure will have introduced some variation in whether a particular mentor was demographically matched or not. In Harris Academy Bermondsey, we use the procedure described as our main procedure above to assign treatments. However, in both Harris Academy and Queen Elizabeth’s, we are unable to collect some belief updating outcomes due to technical issues with our survey. We thus plan to (a) exclude both Harris Academy Bermondsey and Queen Elizabeth’s Grammar School Faversham from analysis of belief updating outcomes, and only use their data for analysis of applications and attendance outcomes; (b) only include Queen Elizabeth’s in comparisons where arms 1b and 2b are pooled with each other (i.e. those that do not distinguish between demographically matched and unmatched mentors).

An earlier version of this pre-registration described the intended design for our first wave in autumn 2024, which did not feature the demographic match treatment variation: this design had arms C, T1 and T2, where T1 corresponded to pooling T1a and T2a from the current design, and T2 corresponded to pooling T2a and T2b from the current design. Due to logistical constraints, we were not able to offer subsidized visits as expected in the first wave. The current version of the pre-registration describes the design for the upcoming second wave in summer 2025. Data from the first wave will be analysed on the basis of the initial pre-registration, while data from the second wave will be analysed on the basis of the current pre-registration; the second wave is still in development. At the time of this update, we had not yet collected final application outcomes from the first wave.

Randomization will be conducted on a computer. We randomize students between arm C and the treatment arms when they complete the baseline survey using randomization in Qualtrics, stratifying by school. As noted above, for Queen Elizabeth’s Grammar School Faversham, randomization in the baseline survey only randomized between arm C and a treatment arm providing both mentors and visits without distinctions between demographically matched and unmatched mentors.

Individual, stratified by school

No

As the randomization is at the individual level, the number of clusters at the unit of randomization level is the same as the number of individuals. However, we will be sampling particular schools, and our central estimate is that we will recruit 20 schools.

Our central estimate is that we will have a sample size of 2000 individuals, but we may get fewer or more as we are still in the process of recruiting schools to the study and confirming participation.

Subject to budgetary and logistical constraints, we plan to assign 50% of participating students to the active control group, and evenly assign students to each of the four treatment arms, so that 12.5% of students are in each of arms T1a, T1b, T2a and T2b. Given our central estimate of 2000 participating students, this implies that we will have 250 students in each treatment arm and 1000 in the control arm. If we encounter constraints on our budget or the capacity of our mentors, we will reduce assignment to treatment evenly across the four treatment arms.

Students in the control group C will be notionally assigned (for the purpose of constructing placebo outcomes) to one of the four treatment groups in even proportions; under our main plan this will mean notionally assigning them to each of the four groups with 12.5% probability.

We conduct power calculations via simulation for our main binary outcome of applications to university on the basis of the following assumptions: - The outcome has a baseline probability of 20% (this seems like a reasonable estimate for the specific universities that students are assigned) - We have 2000 participating students and assign each treatment with probability 12.5% at each school - We allow for positive spillovers between a students’ three best friends that they report in a school in the data-generation process - We regress the outcome on dummies for the treatment (or relevant pooled treatments) and a linear control for the number of friends treated (from 0-3) Under these assumptions, MDES are as follows for the following comparisons of interest, including some pooled comparisons: - T** vs. C (treatment effect of any given treatment arm): 8.5pp - [T2a + T2b] vs. [T1a + T1b] (marginal effect of demographic match): 8.5pp - [T1b + T2b] vs. [T1a + T2a] (marginal effect of visits): 8.5pp - [T1a + T1b + T2a + T2b] vs. C (pooled effect of any treatment): 5.1pp - [T2a + T2b] vs. C (effect of visits relative to control): 7.0pp