Testing Recommender System with jobseekers in France

Our team has been working for several years with the French PES to develop an algorithm to suggest job offers to job seekers. This algorithm, called VADORE, individually ranks available vacancies based on the likelihood of recruitment using data from past hires. It differs from the matching system (SDR) currently used by the French PES, which is an expert system that recommends vacancies based on the proximity between the jobseeker's search criteria, and the characteristics of the vacancies.
We have performed a series of beta tests in which job seekers were occasionally exposed to vacancies through a single mailing. These tests have helped refine the algorithm. The beta-test also showed that job seekers valued the recommendations generated by VADORE at least as much as those provided by the SDR, suggesting a real demand for this type of recommendation.

The intervention that will be tested here correspond to exposing jobseekers for a long time to flows of vacancies generated either by VADORE or by the SDR. Jobseekers will be randomly chosen to receive recommendations from either the AI or the Criterion algorithms. They will first receive a notification email, which, among other things, indicates the motivation for sending recommendations and emphasizes the lack of connection with their job search obligations. Jobseekers will then receive a weekly email with 10 jobs recommended by the selected algorithm. These jobs are selected as follows: 5 will be the best of all available vacancies, and 5 will be among the best jobs they have not seen in previous mailings. The email will display various information about each recommended job: the title (i.e., the job), the salary, the location, the hours, and the type of contract.

2025-02-19

2025-12-31

Job search 1 - Number of applications or clicks since the beginning of the experiment until a job is found.
Job search 2 - Number of applications or clicks since the beginning of the experiment until a job is found weighted by the probability that there is a hire

Hire - 1 In an employment spell during the considered week
Hire - 2 In an employment spell that started with a hire on a recommended (or potentially recommended vacancy) during the considered week
Hire - 3 In an employment spell during the considered week weighted either using an index of the match between the job characteristics and the job seeker's search criteria (wage, occupation, skills, experience, location, contract, working hours) or using a suggestive (possibly predicted) index of the perceived quality of the job by the jobseeker (see explanation below)
Hire - 4 In an employment spell that lasted more than 6 months during the considered week
Hire - 5 Unemployment duration
Hire - 6 Employment duration
Hire - 7 Labor earnings over the period
Hire - 8 Unemployment benefits received until finding a job
Hire - 9 Earnings sum of the two former variables

As part of our VADORE algorithm we are able to compute the probability that there is a match. This is what will be used in primary outcome 3
We intend to run short surveys for each jobseeker who found a job asking about their satisfaction regarding several aspects of the job and then their overall assessment. We will use these data to predict for each jobseeker the value of a vacancy. This is what will be used for primary outcome Hire 2.

W - Jobsearch - 1 (W stands for weekly) Clicks and applications on recommended (or potentially recommended) vacancies over each week period.
W - Jobsearch - 2 Clicks and applications on vacancies posted at the public employment over each week period.
W - Jobsearch - 3 Probability of a match for vacancies on which jobseekers clicked or applied over each week period.
W - Jobsearch - 4 Value of the vacancy on which jobseekers clicked or applied over each week period.
This is for jo seekers who are still unemployed, so this involves differential selection in the different treatment group. This will have to be addressed. One potential solution is to use DDML.


Competition - 1 number of clicks and applications by other jobseekers on vacancies that were recommended to the jobseeker and on which he/she clicked or applied
Competition - 2 number of clicks and applications by other jobseekers on any vacancies on which the job seeker clicked or applied
Here also this is for jobseekers who are still unemployed. Again this involves differential selection between groups that have to be addressed, for example again using DDML


Vacancies
For each vacancy posted at the public employment service we will define several outcome variables
V_Search - 1 (F stands for firm) # Clicks and applications on vacancies posted at the PES over each one week period,
V_Search - 2 estimated probability of a match on weekly click sets and application sets for all vacancies

V_Hire - 1 the vacancy was filled before the considered week
V_Hire - 2 the vacancy was filled before the considered week weighted with an index of match with job search characteristics available (occupation, skills, experience)
V_Hire - 3 duration to fill the vacancy
Here also this is for vacancies who are posted and still open. As applications and filling rates depend on treatment assignment of jobseekers, this might vary by type of treatment. There might also be differential selection between micro-labor markets that have to be addressed, for example again using DDML

The objectives of this experiment are as follows:
Measure the impact of regular recommendations sent by email: We plan to send a weekly email to job seekers offering them the top 10 jobs currently available based on various criteria: 5 will be the best jobs overall, and 5 will be selected from jobs not recommended in previous mailings. This mailing will take place over a long period of time.
Measure the impact of congestion on the exposure of specific job seekers: Examining recommendations based on past data reveals several points. First, in the absence of recommendations, labor markets are congested: a small number of vacancies concentrate the majority of applications. The introduction of algorithmic recommendations may exacerbate this congestion. Thus, some of the potential gains associated with recommending jobs where jobseekers have the best chance of being hired are eliminated by intensifying competition among jobseekers. We therefore want to use a design that allows us to measure the magnitude of this congestion effect and the intensity with which it reduces the pure effect of recommendations.

The experiment is carried out with jobseekers of categories A, B and C. Category A corresponds to jobseekers who are required to carry out positive acts of job search and who have been unemployed during the month. Category B (or C) includes people who: 1) are registered with France Travail, 2) have had a reduced activity of up to (or more than) 78 hours during the last month, and 3) are looking for a CDI (indefinite duration contract), CDD (short-term contract), seasonal or temporary job.

The experiment uses double randomization. The experiment will be conducted at the regional level in Auvergne Rhône Alpes (ARA). The experimental design uses double randomization. First, micro markets are randomly assigned to one of the 4 possible types m. Micro markets of type m=0 serve as a control group. Then, micro markets m=1,2 will receive recommendations from the VADORE recommendation algorithm with two different exposure levels: low (25%) and high (90%). Jobseekers in micro-markets m4 will receive recommendations generated by the SDR with a low exposure rate (25%).
For each micro market, job seekers who will receive vacancies through these systems will be randomly selected according to the planned exposure proportions (25% or 90%).

Building micro markets. Elementary micro-markets are created by crossing occupations, classified according to the ROME (Répertoire opérationnel des métiers et des emplois) nomenclature (114 categories), with employment areas defined by the BMO (Besoins en Main-d’Oeuvre) nomenclature, which distinguishes 80 such areas within the Auvergne-Rhône-Alpes (ARA) region. These micro-markets are then aggregated based on a similarity matrix, constructed from recommendations on the full sample of job seekers one year before randomization and the flow of application to FT vacancies over one year as well. The similarity matrix is obtained from the application data for all job seekers in the previous year. To build this matrix, we compute the matrix P of the share in all applications made by the full set of jobseekers for (respectively to) jobs in micromarket j made to (respectively by) job seekers registered as searching in micro market i. The similarity matrix we consider is then PP’, as it describes the competition job-seekers from a micro-market i face on all micro-markets from job-seekers from micro-market j. Aggregation will be performed so as to minimize the flow of applications or recommendations outside a micro market. This means that, ideally, each micro market would operate autonomously, so that interactions are within a micro market and not between micro markets. There are 5400 elementary micro markets turned to a number of 3060 aggregated micro markets.

Quadruplets of micro markets. Once the micro markets are constituted, they will be assigned to 765 quadruplets. This assignment is done so as to minimize the sum of the differences within a quadruplet of micro market level variables: 1) the number of jobseekers registered in the micro market a month before the start of the experiment, 2) the number of potential competitors job seekers in this market are facing, 3) the share of jobseekers with at least one application to a vacancy posted at FT, 4) the number of posted vacancies for jobs in the micro market (one month before the start of the experiment) 4) the average VADORE score of recommendations made to jobseekers registered in the micro market, 5) the share of women registered in the micro market and 6) the share of Category A jobseekers registered in the micro market. To compute the distance we use a Mahalanobis distance penalized to give less weight to differences regarding gender and category A jobseekers.

Assignment of micro markets: Once the quadruplets are built we randomly assign within each quadruplet one micromarket to each of the possible assignments (control, Vadore Low, Vadore High, Sdr Low). This assignment of micro markets is final and is done before the start of the experiment.

Selection of job seekers and random assignments. All category A, B and C of job seekers residing in the ARA region, having a contractualized Offre Raisonnable d’Emploi (i.e. the parameters of the job search have been registered and validated during a meeting with a dedicated caseworker), having consented to receive emails from France Travail, and aged over 18 will be included in the experiment. Two populations are distinguished:
"Stock" population: it includes category A, B and C job seekers aged 18 or over and registered with France Travail the week before the start of the experiment. These job seekers are part of a micro market "m". If in this micromarket a non-zero proportion is assigned to receive recommendations with an assignment probability. They are then randomly assigned to receive recommendations specific to their micromarket (Vadore: m = 1, 2; SDR: m = 3) according to this probability. Exactly a proportion of these job seekers registered at the time of the experiment's launch will be drawn to be assigned to the corresponding treatment. The assignment is stratified by gender and by Category A versus (B and C) status.
"Flow" population: It includes category A, B and C job seekers aged 18 or over registered with France Travail since the launch of the experiment. Their date of inclusion in the experiment coincides with the date of their meeting with a dedicated caseworker to define the ORE. The assignment is also stratified by gender and category A status. For this purpose, in each treated micro-market there are 4 assignment lists, one for each modality of crossing of the gender and category characteristics. These assignment lists are a stack of quadruplets containing three Cs and one T for m=1 or 3, and one C and three Ts for m=2, randomly arranged. When a jobseeker enters the experiment, he or she is assigned a micro-market and a type, which are linked to the jobseeker's gender and status characteristics. We then consider the rank of his registration time among the jobseekers of his type registered in his micro-market. The observation of the same rank in the assignment list of his type defines his status in terms of treatment.

The important dimensions of heterogeneity that we will examine are related to both market characteristics:
- The proportion of jobseekers with at least one application for a vacancy submitted to the PES;
- Market tightness
- Quality of Vadore recommendations.

And jobseekers characteristics:
- gender,
- JS registered under the status of categories.

Not available

Randomisation done in office by a computer

Double randomization: micro markets are first randomly assigned from clusters of 4 micro markets. Jobseekers are then randomy assigned according to the treatment status of the micro market.

Yes

765

500,000 (the experiment involve recruitment form the flow, it is not possible thus to give a precise number) The initial stock is 300,000 but there are monthly inflows

765 micro market control 765 micro market with 25% exposition to VADORE (31250 JS) 765 markets with 90% exposition to VADORE (112500 JS) 765 micro markets with 25% exposition to SDR (31250 JS)