Intervention(s)
We study the implementation of job recommender systems for jobseekers at the French PES. For each jobseeker, an initial algorithm ranks job ads according to their estimated hiring probability, using a state-of-the-art machine learning algorithm trained on past hires recorded by the PES. We refer to this version as V.0.
Our first beta test [registered as https://doi.org/10.1257/rct.8998-1.3] compared the performance of recommendations generated by V.0 with (i) those from an algorithm based on the match between job ads and search criteria (S) and (ii) those based on a combination of the rankings from these two polar systems, V.0 and S.
This first beta test showed the value of combining information from both rankings. It led to the development of a new algorithm, V.2, which introduced several innovations (For a comprehensive exposition, see our paper: Bied, G. and al. (2023). Toward job recommendation for all. Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence (IJCAI), AI for Good , 5906–5914.
). First, the algorithm’s architecture was modified, introducing a two-step procedure, in which the top 1000 jobs first selected by V.0 are re-ranked by a more expressive neural model in a second stage. Second, the components used in S were incorporated as predictors in the second-stage hiring model. Third, part of the second-stage model was dedicated to predict job seekers’ applications (algorithm A see below); the resulting application probability score was itself then used as a component in the prediction of hiring probability.
We also considered the algorithm (A) in isolation. This algorithm follows the same learning structure as V.0 and the second stage akin to V.2’s but with the purpose to predict applications.
The main goal of the current beta test is to compare the performance of V.0 and S with that of V.2 and A. However, we also include a new algorithm combining the rankings of V.2 and S—similar to the first beta test—referred to as Mixture.
In addition, we include an algorithm trained with XGBOOST, which represents the state of the art, while the other machine learning algorithms are trained as neural networks.
A final feature of this beta test is the inclusion of recommendations from an expert system, S*, currently used by the PES. Our version S is only an approximation. The S* algorithm also applies numerous filters, which means it does not necessarily return offers for every jobseeker.
The algorithms compared are therefore: S*, S, V.0, V.2, A, Mixture, and XGBOOST.
Additional treatment arms consider recommendations from the V.2, A and Mixture algorithm (renamed V.2.I0, A.I0 and Mixture.I0), to which varying levels of explanation are cumulatively added:
The first level focuses on the purpose of the algorithm (V.2.I1, A.I1 and Mixture.I1),
The second on its performance (V.2.I2, A.I2 and Mixture.I2),
The third on the rationale behind its relevance (V.2.I3, A.I3 and Mixture.I3).
The experiment proceeds as follows:
Jobseekers receive an invitation email.
By clicking on a link in the email, they access an online survey.
Jobseekers are first shown five recommended vacancies and asked to evaluate the overall relevance of the recommendations.
After rating these five vacancies, they reach a final page displaying the same five ads, along with five additional recommendations. They are not required to rate the five additional vacancies. On this final page, they can click on job ads to view details and apply.