Managerial implicit stereotypes and where to find them: Evidence from Incentivized Resume Rating

Recent literature has shown that if individuals become aware of their stereotypes, they change their behavior. We want to study whether managerial implicit gender bias lead to discriminating behaviour towards women. Moreover, we want to study whether making managers aware of their implicit bias induce them to change their behaviour when evaluating CVs. Through the new approach proposed by Kessler et al. (2019), we exploit the Incentivized Resume Rating to elicit managers preferences in terms of characteristics needed to become a promising manager. We deliver a survey to a large sample of managers, who are asked to evaluate explicitly fake CVs, constructed on the basis of real CVs of Bocconi SDA’s master students. Managers who will respond to our survey will hence be aware that the CV they are going to evaluate are fake (but still realistic): this prevent our study from any type of deception. The experiment is made of three steps. In the first step we will deliver a first survey in which we will ask managers to provide demographic characteristics, to express their beliefs and attitutes toward gender-related topics, to provide information of their firm’s performance and to complete the IAT test Gender-Career, with which we will assess their implicit gender bias. We will then divide the sample of managers into two groups, treatment and control. In the second step of our experiment we will communicate to the traeatment group of managers their IAT score. In the third step we will deliver to both treatment and control group the Incentivized Resume Rating, randomizing the timing of delivery in order to study whether the effect of the revelation of the IAT are persistent over time. We expect managers informed of their stereotypes before the IRR to provide higher scores to profiles of women.

We are collaborating with Federmanager, the biggest Association of managers in Italy. We will deliver the survey through Federmanager’s channels.
In particular, Federmanager has access to email addresses of its subscribers. To ensure data privacy, Bocconi researcher will be creating unique Qualtrics survey links that will be provided to Federmanager. Federmanager will send these links to their subscribers. Bocconi researcher will hence not be able to match answers to email addresses or names of respondents, as these will not be provided to Bocconi researchers. Federmanager, on the other hand, will receive only aggregate results of the survey answers. This is done to ensure complete anoymity of managers responding to the survey.

The procedure is the following:
1. Bocconi researcher create N unique links associated to N identifying numbers
2. Federmanager will receive these links with the associated identifying number and will match them with individual subscribers and send the survey.
3. Only a small fraction n of N will respond to the survey.
4. Bocconi researcher will receive the n answer to the survey, with their associated link. Bocconi researchers will hence have a dataset of answers associated to the n remaining identifying numbers.
5. Bocconi researcher will be able to stratify the sample of n respondents into treatment and control group.
6. Bocconi researcher will create other survey links and will associate them to those same n numbers for the second and third steps of the experiment (respectively the communication of the IAT score and the survey containing the IRR).
7. Federmanager will match again those new links associated to the n identifying numbers to the email addresses of point 2 and send the new links.

Our other partner is the Bocconi SDA Executive Master. Students from the Executive master will be asked to provide their anonymous CV so that we will be able to construct the fake profiles that will be delivered to managers for evaluation. Bocconi researcher will hence not be provided any information on personal data of students who are willing to take part to the project, such as name or day of birth.
The previous passage is extremely important for our incentive strategy
In particular, the Incentivized Resume Rating establish that respondent are aware of the fake nature of the CV they are evaluating, in order to avoid deception. However, these CVs have to be realistic in order to provide the right incentive to managers to respond with attention. In particular, we will ask managers whether they want to receive in the future real CVs of students that mostly resembles the profiles they scored the highest during the Incentive Resume Rating. After the experiment, we will ask Bocconi SDA Executive Master students whether they want to provide their CVs to be sent to those managers that would probably like their profile.
As already stated, all correspondence with the managers will be done through Federmanager, which will be provided the unique Qualtrics links through which we deliver all survey steps and communication to managers.
So, in case managers are willing to receive real CVs in the future and students are willing to provide their CVs to be sent to managers, we will be sending CVs through the same procedure as above described.
Another source of incentive to managers will be given by telling them that their profile evaluation will help the SDA Executive Master program to identify the most promising “managers of the future” among the students. We will provide to the Executive Master Program the characteristics preferred by managers, and students will be selected by the Master Program itself for participating in a meeting organized jointly by Bocconi SDA and Federmanager for presenting the association to students who will likely become managers in the future.

The experiment consists in administering a survey to Italian managers, including an Implicit Association Test to capture the implicit stereotypes that associate women and family against men and career. The survey is conducted from personal computers and will be accessible through unique links that managers will receive directly by email. Managers who agree to take part in the survey give written informed consent by confirming their intent to participate at the beginning of the survey. The time to complete the survey is around 30 minutes, and participants do not receive any compensation.
As already stated, the intervention aims at making managers aware of their implicit bias by revealing the score of their IAT test. Please find in the attachments the text of the communication they will receive.
We will ask to all the sample of managers whether they want to receive feedback on the IAT score after taking the test.
To the treatment group, the score will be communicated before the second survey containing the Incentivized Resume Rating. To the control group, the score will be communicated straight after.
The feedback will provided over e-mail (treatment group managers will receive a new unique Qualtrics link which will display their own IAT score). Together with the IAT score, it will also be provided a brief description of the test explaining whether in their case the association between women and family (or men and career) is “slight”, “moderate” or “strong” ased on the thresholds typically used in the literature (Greenwald et al., 2009). They will also be reassured that these results would not be shared with anyone.
The incentivized resume rating will be sent through a second Qualtrics survey and will be delivered one or three weeks after the disclosure of IAT score to the treatment group.
To see the text of the first and the second survey, please check the attachments. The attachments also include the messages that managers receive for the presentation of both surveys and the text of the message sent to reveal the IAT score.

We will also provide another, independent, treatment, in which we will provide to a second treatment group an informative treatment in which we provide explanations of how the labour market is affected by the gender gap and how managerial gender bias can affect decisions: this is a cost effective treatment, and will be useful to compare other ways to debias managers with our preferred treatment of revealing stereotypes.

As explained, the experiment is done on two differente populations to increase our sample: the first is the one of Federmanager, the second is the one of a high tech company, Leonardo S.p.A.

With Leonardo S.p.A. the procedure is easier. A Qualtrics link contaning the first survey is sent directly to each employee of the company by the HR team. At the end of the first survey, respondents are asked whether they want to leave their email address to be contacted again in order to receive their IAT score.

After one week from the end of the first survey, all Leonardo S.p.A.'s respondents who left their email would be recontacted. They will be given the possibility to either receive their IAT score or to also participate in a second survey entailing the evaluation of 10 CVs following the methodology of IRR. The treatment group would receive the IAT score before starting the evaluation, the control group would receive the IAT at the end of the CVs evaluation. What we have been calling the Placebo group, instead, would receive the information about the gender gap in the labor market before starting the CV evaluation.

2022-01-15

2022-01-30

The primary outcomes will be the score provided by respondents to the Incentivized Resume Rating. Through the Incentivized Resume Rating we will ask managers to evaluate 10 explicitly fake CVs, that contain a bunch of information such as gender, GPA, previous education, work experience, experience abroad, languages.
Please find attached the second survey containing the IRR. You will see only one CV, as characteristics inside of it are randomized for 10 iterations, while keeping the shape and presentation of the CV fixed. At the end of each CV, a slider will be used to give an evaluation from 1 to 10.


Managers will be asked to provide an evaluation of each profile on a Likert Scale from 1 to 10. We are interested in understanding how the gender of the profile and manager’s own implicit bias affect the score provided to the profile. Other variables included in the profile and manager’s characteristics will be used as control variables. Our preferred specification will be as follows:

〖Profile Score〗_im= α+βX_i+ρ〖Gender〗_i+ γ〖Treatment〗_m×〖Gender〗_i+δ_m+ε_im

Where X_i are profile characteristics, δ_m are manager’s m fixed effects and γ is our coefficient of interest, identifying the effect of IAT score disclosure on on outcome variable 〖Profile Score〗_im.


We replicate the study after 8 months, at the end of January 2023, in order to study whether there is persistency of the effect of the treatment. This time, both Treatment and Control group are "treated", in fact Control group received the IAT score at the end of the first survey in which they were asked to evaluate the Curricula through the incentivized Resume Rating. We hence expect no difference between Treatment and Control groups in the long run (as they are both treated now), but still expect better results in terms of diminished gender gap in scoring with respect to the results obtained with the Control group in the short run. We expect the effect of the treatment diminishes over time, so results might not be as clean and clear as in the short run.

The other outcomes will be given by the IAT score itself, which will be regressed on manager’s individual characteristics.
We will also use firm’s performance variables and the share of women in the firm and in managerial positions as outcome variables, and study whether manager’s bias affect these figures.
Evetually, we also use explicit attitudes and belilefs of managers on the IAT score, showing whether correlation arises between explicit and implicit attitudes. This is ex ante ambiguous and the prior is that the explanatory power of the IAT score in explaining explicit bias is low. This might be due both to Social Desirability Bias, for which managers provide answers that are more acceptable, but also to the fact that managers are trained in answering these type of questions in a non discriminatory way (thanks for example to diversity training they have undertaken during their career).

An additional secondary outcome is the time spent on CVs. This can be recorded through Qualtrics.

In order to incentivize the participation in the second survey, we would also provide the possibility to win a 200 euros prize for those who would finish the resume rating. At the end of the resume rating, we ask both treatment and control group to state how much of these 200 euros, in case of win, they would donate to an association promoting women empowerment in the labor market. This is an incentivized outcome as well and might be interesting if our treatment has any effect on this.

We also test whether in the long run respondents will remember their IAT association. At the end of january 2023, in the replication of the IRR, we will also ask whether a candidate remembers his own score (and ask which one it was, to see if it matches).

The treated grous receive the treatment before the Incentivized Resume Rating (IRR). Managers in the control group will receive the control straight after the IRR.

Please refer to the "hidden" experimental design for more information: excessive details in these section might alter results.

To test the long-run effect, we will also replicate the IRR after about 8 months, in February 2023. We want to test whether there is persistency of the results. We will also ask to respondents whether they remember the score of their IAT and whether thet think knowing their IAT results changed their behaviour. Moreover, we ask whether they think that there was any link between the first survey in which we collected the IAT, and the surveys in which we ask to evaluate the CV through the IRR. This is because we want to overcome demand effect and be sure that the respondents did not understand the purpose of the research.

The timing of the Incentivized Resume Rating will be randomized across managers: in particular, Federmanager will be able to tell us whether two or more identifying numbers as described in the “Partner and survey delivery” section belong to the same firm (this is however implausible and very rare as stated by Federmanager). So, for ease the explanation, let us think as the unit of our analysis the firm, which might have one (or more managers) taking part to the survey. Again, we don’t know neither the identity of the manager nor the identity of the firm. So for firm with only one manager responding to the survey, the manager will correspond to the firm itself.

To be noted is that the randomization will be carried out at manager level if we have no firm with multiple answers (more plausible event). Our prior is that we will not have answers from managers belonging to the same firm, as Federmanager has never seen such a case in previous surveys delivered to its subscribers. But if this is not the case for our experiment, we will choose to randomize at the firm level rather than at the manager’s level in order to avoid contamination between managers who receive the early feedback and those who receive the feedback after the IRR.

The treated grous receive the feedback (IAT or general information treatment) before the Incentivized Resume Rating. Managers in the control group will receive the IAT feedback straight after the IRR. We are obliged to reveal to everyone his/her IAT, as the Bocconi Ethics Committee requested us to do so, in order not to annoy the managers in the control group.

There is no decepetion, managers are aware that CVs are fake.

Collaborating with Federmanager, the biggest association of Italian managers, we aim to deliver a first survey in which we will collect managers' Implicit Gender Bias (IAT Gender). The survey will be distributed directly from Federmanager, hence Bocconi researchers will not have access to mail addresses of respondents (but just to anonymized identification numbers), nor any kind of question that would allow any sort of identification of the respondent will be asked (we will ask demographics questions, such as age, gender, education, but will not ask any information about the name of firm they work for).

The data collection of the first survey will last two weeks, starting from end of October.
After ten days (during which the researchers will divide respondents into treatment and control group), the treatment group will receive the communication about own IAT score. After one or three weeks, we will be sending a second (experimental) survey to both treatment and control groups, in which they will be asked to evaluate fake profiles (CVs) of MBA students and will ask to give an evaluation on a Likert Scale for the extent to which they think that profile is "promising to become a good manager".

Outcomes (dependent variables) will be the score (Likert from 1 to 10) of each profile evaluated by respondents. Independent variables will range to the treatment variable (IAT score revealed or not), IAT score, managers' demographics, profile's characteristics (gender, age, field of specialization, ..).

In order to construct realistic CVs of Master's student, we will be collaborating together with Bocconi SDA. MBA students will be asked whether they are interested in participating to this project and, in case, to provide us their CVs. We would scrape their CV and construct fake profiles on basis of true CVs. Characteristics in fake profiles are randomized, as done in Factorial Survey Analysis.

There will be no participant remuneration. The incentive we provide is twofold:
1. we ask managers to contribute to identify the "most promising managers of the future" for a presentation of the Association to Master students. We will contact those students that mostly resemble the preferred fake profiles and ask them whether they would be interested in participating in an event in which Federmanager presents itself.
2. Upon student consent, we will ask managers if they are willing to receive Master students' CVs that mostly resemble their own preferred characteristics.

The randomization will be done in office by a computer. We will stratify managers by their demographic characteristics and firm’s characteristics. In particular, the characteristics we will take into consideration when stratifying the managers into treatment and control group will be:
- Sector: it is in fact well known in the literature that more discriminatory behaviour might affect some sectors more than others
- Gender of the manager
- Age of the manager
- IAT score

We will divide the sample of respondents in 6 groups, 4 treatment and 2 control groups. This will be done to study whether the revelation of the IAT score (the treatment) has persistent effects over time. The first treatments and control group will hence receive the Incentivized Resume Rating after one week from the revelation (of the IAT or of the general information about the gender gap). The second groups will receive the IRR after three months.

The randomization unit will be at the manager's level. As the Association of managers we are collaborating with ensured that respondents never belong to the same firm. Anyway, we will know whether two or more respondents belong to the same company.

If such an event happened, our unit of randomization will be the firm instead of the single manager.

In case of Leonardo S.p.A. the randomization unit is still at employee level. However, we distinguish between Employees and Managers, since these are two different groups and might respond differently to the treatment. Moreover, since Employees are much more than Managers, Employees will be divided in three groups (Control, Treatment and Placebo) as explained above.

No

The sample size has a planned number of cluster equal to 3000.
We will deliver the survey to about 30’000 managers, but previous survey run by the Company we are collaborating with (we do not display the name here publicly for experimental reasons) show that usual take up rate if of about 10%.
Each of the six treatment and control goups is estimated to contain about 500 managers.

We are not sure of the number of Employees and Managers from Leonardo S.p.A. who will respond and who will be willing to participate in the various steps of our experiment.

The planned number of observations is 3000

500 managers: first treatment (revealing IAT and receive IRR after one week: IAT & Short Run effect)
500 managers: second treatment (informative treatment on gender gap and receive IRR after one week: Information & Short Run effect)
500 manager: first control (no treatment and receive IRR together with first and second treatment groups: Control & Short Run effect)

500 managers: third treatment (revealing IAT and receive IRR after three months: IAT & Long Run effect)
500 managers: fourth treatment (informative treatment on gender gap and receive IRR after three months: Information & Long Run effect)
500 manager: second control (no treatment and receive IRR together with third and fourth treatment groups: Control & Long Run effect)

In Leonardo, instead, the treatment is given inside the survey containing the IRR. This is done to reduce the attrition rate in the various steps of the experiment.

The IRR will contain abut 20 profiles that the respondents will have to evaluate. If all the respondents in the first survey will respond to the second survey containing the IRR, we will receive about 12'000 evaluations of CV. Groups of 500 for treatment and control outnumber the ones usually used in the literature.