Pay for applications

2019-09-23

2019-10-14

Key outcome variables include:

The number of applicants (men and women) that complete their assessment tasks in the different treatments.

Assessment task scores (men and women) in the different treatments.

The quality of the applicants that complete their assessment tasks in the different treatments.

Gender difference in the effect that monetary incentive has on the quantity and quality of the applications.

The number of women reaching shortlisting criteria (e.g. top 25% score in the assessment test score) in the different treatments.

We are interested in the number of completed applications (assessment tasks) between treatments, and any gender difference in the control group and in the treatment group, and whether different treatments have different level of gender difference (Male Control - Female Control vs Male Treatment - Female Treatment). Similarly, we are interested in the quality of applications between treatments, which is measured by a candidate’s score in a Python programming test which forms a part of the assessment task (see AEA RCT registry pre analysis titled Understanding Skill Gap for more details on the test and scoring). Using the test score, we will also study whether there are gender differences in treatment effects between and within treatments.

To further compare whether the quality of applicants differs between the treatments, we will compare the difference between the treatment and control group in applicants’ “predicted programming skill” based on characteristics that we measured in the applicant survey. We compare predicted instead of actual test performance, because the incentives may have a direct effect on candidates’ performance. To calculate candidates’ “predicted programming skill” we will i) regress candidates’ python skill on characteristics taken from the applicant survey in the base treatment, ii) save the regression coefficients from this regression, and iii) multiply the characteristics of candidates in all treatments with their characteristics taken from the survey.

We also are interested in the characteristics of applicants with completed applications in general. Characteristics are taken from the applicant baseline survey in stage 1 (e.g. education, years of programming experience; age, risk aversion and competitiveness). Considering both the changes in quantity and quality, we would also like to examine how the characteristics of shortlisted candidates differ according to the treatments. Shortlisting is done according to the Python programming test score.

We will test whether the effect of the incentives depend on candidates’ risk aversion and competitiveness.

We will also interact treatments with aptitude and personality test performance

We will measure candidates’ risk aversion and competitiveness using the candidate survey.

In the experiment there will be a control and treatment randomly assigned in order to test potential instruments that aim to reduce gender differences in STEM fields.

In stage 1, we will post a job ad for a programming job (1 month, 80 hours) across the United States at several major job portals (e.g. joinhandshake.com, dice.com, crunchboard.com, github.com) and ask job applicants to send their CV and fill out a short survey (e.g. years of experience programming). After applications close, in stage 2, we will invite a subset of applicants to do an assessment task (Python programming test and an aptitude test (half of the applicants) or a Python programming test and a personality test). To make sure that there are a similar number of male and female applicants in each treatment, we will invite a random sample of male applicants and a random sample of female applicants to complete the assessment task. The key feature of the experiment is that we will randomly assign all invited applicants to one of two treatments: control, inventive.

In the control treatment, applicants will not be incentivized to complete the assessment task. In the incentive treatment, when applicants are invited to take the assessment, they will be informed that after completion of the assessment task they will be paid a base pay of USD 5 and a pay that depends on their test performance. Total pay will range between 5-10 USD.

Randomization will be carried out by a computer.

The randomization unit will be the individual applicant for all treatments.

No

The number of clusters will equal the number of observations.

We aim to invite 720 applicants (360 male and 360 female), which would give us a maximum number of 720 observations (of these invited applicants, only a subset will complete the assessment test). The actual number of invited applicants depends on the number and gender of applicants.

We aim to have 360 applicants in each treatment group:
• Control: 360 applicants (180 male and 180 female)
• Incentive: 360 applicants (180 male and 180 female)