The effect of informational sessions on student’s college application: evidence from a policy experiment in France

The goal of this intervention is to improve educational track decisions and to assist students, particularly girls and those from disadvantaged backgrounds, in making better-informed decisions about their future studies. A user-friendly online platform called MonProjetSup was designed to help high school seniors explore potential educational pathways by providing personalized recommendations based on their academic background, interests, and preferences. The system incorporates an algorithm that suggests relevant tracks (rather than specific programs), allowing students to discover options they might have otherwise overlooked. A track is a combination of a field of study and a type of diploma (for example, bachelor’s of mathematics, corresponding to an official government classification), whereas a program also specifies the institution of higher education (for example, bachelor’s of mathematics at the University Lyon 1).

2024-01-08

2024-03-14

All measured using students’ university choices in the Parcoursup application platform:
-How diverse applications are
-How ambitious applications are

-Diversity of applications is measured by the number of different tracks, the number of different tracks that were not reported as a track of interest in the first sitting, and the variance in the selectivity of applications.
-Ambition is measured considering the pool of students enrolled in a program, and computing the median high school diploma grades of these students; and that same index relative to the student’s own mark.
-We will explore heterogeneity with respect to : gender, low/high SES, low/high marks.

-Additional measures of ambition
-Dyadic analysis

-Additional measures of ambition are the average admission rate of the programs; the lowest median GPA among the programs to which the student has applied; the highest median GPA among the programs to which the student has applied. We will also compute the same dimensions relative to the student’s own mark (median GPA minus own GPA).
-Dyadic analysis will estimate the probability that student i applies to field j, when field j has been recommended to him/her, compared to when it hasn’t. This can be achieved by computing the probability of recommendation (i,j) conditional on the information filled during the first sitting, and using the Horvitz-Thompson estimator to compare the outcome of students in the treatment group that have been recommended j with students in the control group that haven’t, but have the same probability to.

The main experiment (experiment 1) tests the effects on high school students’ higher education (HE) track choices, of providing information and individualized recommendations on HE tracks. Randomization happens at the class level in a set of volunteer high schools. In a first sitting, all classrooms’ students were offered a prototype of the MonProjetSup platform, asking them questions about their history, preferences and areas of interest, and offering relevant suggestions of careers that might be of interest to them. Randomization of classes followed the closure of that sequence; it was stratified by high school and within high schools, classes regrouped by majors, and restricted to classes that had participated in the first sitting. A second sitting was then reserved to treated classrooms, during which students were offered precise and individualized recommendations for tracks of study, determined by an algorithm that used the information filled-in during the first sitting. This intervention takes the form of an encouragement design as professors and students were free to participate in the second sitting and focussed with varying degrees of intensity on the information provided to them. However, students in the control group did not at any point have access to the treatment. To increase take-up, we also allowed treated students to participate in the second sitting at home if they had not done so at school.
The nested experiment (experiment 2) consists in an encouragement design testing the effect of providing information on programs’ selectivity. It was conducted in the subset of classes that were assigned treatment in experiment 1. For this A/B test, half of students who opened an information note on a track were readily given information on the distribution of grades in the target program, and the other half had to exert effort to retrieve this information. Treated students were determined ex ante by a Bernouilli trial (at rate 50%).

Computer randomization. For assignment of classes to control or treatment in experiment 1: stratified by high school and then by major. For assignment of students to control or treatment in experiment 2: no stratification.

Classes for experiment 1, students for experiment 2.

Yes

207 recruited classes

5056 students

In experiment 1 :105 classes assigned to treatment; 102 classes assigned to control. In experiment 2 : 376 students for Version A (test), 384 students for Version B (control)

Given a hypothesis of a 0.1 inter-cluster correlation within clusters of classes, our ITT model for experiment 1 would have a minimum detectable effect of 0.14 standard deviations (Cohen’s d) (significance at 5% and power at 80%). There can be several notions of take-up: using the proportion who connected to the platform on the second sitting (50%), this would represent a rescaled LATE MDE of 0.29 SD; using the proportion who actually looked at suggestions (32%), the LATE MDE would be 0.44 SD .For experiment 2, our ITT model would have an MDE of 0.20 SD, with a slightly inflated value of 0.23 SD for a LATE estimation due to the high rate of take-up. ICC is not an issue in this case since treatment is not clustered. Those estimates are upper bounds on the MDE however, since controlling for student-level covariates, or applying the methodology proposed in Hazard & Löwe, may improve statistical power.