Managerial implicit stereotypes and where to find them: Evidence from Incentivized Resume Rating

2022-01-15

2022-01-30

The primary outcomes will be the score provided by respondents to the Incentivized Resume Rating. Through the Incentivized Resume Rating we will ask managers to evaluate 10 explicitly fake CVs, that contain a bunch of information such as gender, GPA, previous education, work experience, experience abroad, languages.
Please find attached the second survey containing the IRR. You will see only one CV, as characteristics inside of it are randomized for 10 iterations, while keeping the shape and presentation of the CV fixed. At the end of each CV, a slider will be used to give an evaluation from 1 to 10.


Managers will be asked to provide an evaluation of each profile on a Likert Scale from 1 to 10. We are interested in understanding how the gender of the profile and manager’s own implicit bias affect the score provided to the profile. Other variables included in the profile and manager’s characteristics will be used as control variables. Our preferred specification will be as follows:

〖Profile Score〗_im= α+βX_i+ρ〖Gender〗_i+ γ〖Treatment〗_m×〖Gender〗_i+δ_m+ε_im

Where X_i are profile characteristics, δ_m are manager’s m fixed effects and γ is our coefficient of interest, identifying the effect of IAT score disclosure on on outcome variable 〖Profile Score〗_im.


We replicate the study after 8 months, at the end of January 2023, in order to study whether there is persistency of the effect of the treatment. This time, both Treatment and Control group are "treated", in fact Control group received the IAT score at the end of the first survey in which they were asked to evaluate the Curricula through the incentivized Resume Rating. We hence expect no difference between Treatment and Control groups in the long run (as they are both treated now), but still expect better results in terms of diminished gender gap in scoring with respect to the results obtained with the Control group in the short run. We expect the effect of the treatment diminishes over time, so results might not be as clean and clear as in the short run.

The other outcomes will be given by the IAT score itself, which will be regressed on manager’s individual characteristics.
We will also use firm’s performance variables and the share of women in the firm and in managerial positions as outcome variables, and study whether manager’s bias affect these figures.
Evetually, we also use explicit attitudes and belilefs of managers on the IAT score, showing whether correlation arises between explicit and implicit attitudes. This is ex ante ambiguous and the prior is that the explanatory power of the IAT score in explaining explicit bias is low. This might be due both to Social Desirability Bias, for which managers provide answers that are more acceptable, but also to the fact that managers are trained in answering these type of questions in a non discriminatory way (thanks for example to diversity training they have undertaken during their career).

An additional secondary outcome is the time spent on CVs. This can be recorded through Qualtrics.

In order to incentivize the participation in the second survey, we would also provide the possibility to win a 200 euros prize for those who would finish the resume rating. At the end of the resume rating, we ask both treatment and control group to state how much of these 200 euros, in case of win, they would donate to an association promoting women empowerment in the labor market. This is an incentivized outcome as well and might be interesting if our treatment has any effect on this.

We also test whether in the long run respondents will remember their IAT association. At the end of january 2023, in the replication of the IRR, we will also ask whether a candidate remembers his own score (and ask which one it was, to see if it matches).

The treated grous receive the treatment before the Incentivized Resume Rating (IRR). Managers in the control group will receive the control straight after the IRR.

Please refer to the "hidden" experimental design for more information: excessive details in these section might alter results.

To test the long-run effect, we will also replicate the IRR after about 8 months, in February 2023. We want to test whether there is persistency of the results. We will also ask to respondents whether they remember the score of their IAT and whether thet think knowing their IAT results changed their behaviour. Moreover, we ask whether they think that there was any link between the first survey in which we collected the IAT, and the surveys in which we ask to evaluate the CV through the IRR. This is because we want to overcome demand effect and be sure that the respondents did not understand the purpose of the research.

The timing of the Incentivized Resume Rating will be randomized across managers: in particular, Federmanager will be able to tell us whether two or more identifying numbers as described in the “Partner and survey delivery” section belong to the same firm (this is however implausible and very rare as stated by Federmanager). So, for ease the explanation, let us think as the unit of our analysis the firm, which might have one (or more managers) taking part to the survey. Again, we don’t know neither the identity of the manager nor the identity of the firm. So for firm with only one manager responding to the survey, the manager will correspond to the firm itself.

To be noted is that the randomization will be carried out at manager level if we have no firm with multiple answers (more plausible event). Our prior is that we will not have answers from managers belonging to the same firm, as Federmanager has never seen such a case in previous surveys delivered to its subscribers. But if this is not the case for our experiment, we will choose to randomize at the firm level rather than at the manager’s level in order to avoid contamination between managers who receive the early feedback and those who receive the feedback after the IRR.

The treated grous receive the feedback (IAT or general information treatment) before the Incentivized Resume Rating. Managers in the control group will receive the IAT feedback straight after the IRR. We are obliged to reveal to everyone his/her IAT, as the Bocconi Ethics Committee requested us to do so, in order not to annoy the managers in the control group.

There is no decepetion, managers are aware that CVs are fake.

Collaborating with Federmanager, the biggest association of Italian managers, we aim to deliver a first survey in which we will collect managers' Implicit Gender Bias (IAT Gender). The survey will be distributed directly from Federmanager, hence Bocconi researchers will not have access to mail addresses of respondents (but just to anonymized identification numbers), nor any kind of question that would allow any sort of identification of the respondent will be asked (we will ask demographics questions, such as age, gender, education, but will not ask any information about the name of firm they work for).

The data collection of the first survey will last two weeks, starting from end of October.
After ten days (during which the researchers will divide respondents into treatment and control group), the treatment group will receive the communication about own IAT score. After one or three weeks, we will be sending a second (experimental) survey to both treatment and control groups, in which they will be asked to evaluate fake profiles (CVs) of MBA students and will ask to give an evaluation on a Likert Scale for the extent to which they think that profile is "promising to become a good manager".

Outcomes (dependent variables) will be the score (Likert from 1 to 10) of each profile evaluated by respondents. Independent variables will range to the treatment variable (IAT score revealed or not), IAT score, managers' demographics, profile's characteristics (gender, age, field of specialization, ..).

In order to construct realistic CVs of Master's student, we will be collaborating together with Bocconi SDA. MBA students will be asked whether they are interested in participating to this project and, in case, to provide us their CVs. We would scrape their CV and construct fake profiles on basis of true CVs. Characteristics in fake profiles are randomized, as done in Factorial Survey Analysis.

There will be no participant remuneration. The incentive we provide is twofold:
1. we ask managers to contribute to identify the "most promising managers of the future" for a presentation of the Association to Master students. We will contact those students that mostly resemble the preferred fake profiles and ask them whether they would be interested in participating in an event in which Federmanager presents itself.
2. Upon student consent, we will ask managers if they are willing to receive Master students' CVs that mostly resemble their own preferred characteristics.

The randomization will be done in office by a computer. We will stratify managers by their demographic characteristics and firm’s characteristics. In particular, the characteristics we will take into consideration when stratifying the managers into treatment and control group will be:
- Sector: it is in fact well known in the literature that more discriminatory behaviour might affect some sectors more than others
- Gender of the manager
- Age of the manager
- IAT score

We will divide the sample of respondents in 6 groups, 4 treatment and 2 control groups. This will be done to study whether the revelation of the IAT score (the treatment) has persistent effects over time. The first treatments and control group will hence receive the Incentivized Resume Rating after one week from the revelation (of the IAT or of the general information about the gender gap). The second groups will receive the IRR after three months.

The randomization unit will be at the manager's level. As the Association of managers we are collaborating with ensured that respondents never belong to the same firm. Anyway, we will know whether two or more respondents belong to the same company.

If such an event happened, our unit of randomization will be the firm instead of the single manager.

In case of Leonardo S.p.A. the randomization unit is still at employee level. However, we distinguish between Employees and Managers, since these are two different groups and might respond differently to the treatment. Moreover, since Employees are much more than Managers, Employees will be divided in three groups (Control, Treatment and Placebo) as explained above.

No

The sample size has a planned number of cluster equal to 3000.
We will deliver the survey to about 30’000 managers, but previous survey run by the Company we are collaborating with (we do not display the name here publicly for experimental reasons) show that usual take up rate if of about 10%.
Each of the six treatment and control goups is estimated to contain about 500 managers.

We are not sure of the number of Employees and Managers from Leonardo S.p.A. who will respond and who will be willing to participate in the various steps of our experiment.

The planned number of observations is 3000

500 managers: first treatment (revealing IAT and receive IRR after one week: IAT & Short Run effect)
500 managers: second treatment (informative treatment on gender gap and receive IRR after one week: Information & Short Run effect)
500 manager: first control (no treatment and receive IRR together with first and second treatment groups: Control & Short Run effect)

500 managers: third treatment (revealing IAT and receive IRR after three months: IAT & Long Run effect)
500 managers: fourth treatment (informative treatment on gender gap and receive IRR after three months: Information & Long Run effect)
500 manager: second control (no treatment and receive IRR together with third and fourth treatment groups: Control & Long Run effect)

In Leonardo, instead, the treatment is given inside the survey containing the IRR. This is done to reduce the attrition rate in the various steps of the experiment.

The IRR will contain abut 20 profiles that the respondents will have to evaluate. If all the respondents in the first survey will respond to the second survey containing the IRR, we will receive about 12'000 evaluations of CV. Groups of 500 for treatment and control outnumber the ones usually used in the literature.