What Drives Different Assessments of Performance on Coding Tasks?

We aim to assess how software developers evaluate pieces of code written by computer programmers. Our experiment uses a large set of de-identified code blocks from an online coding platform, which span coders of different skill and problems of different levels of difficulty. For each code block, we will have access to the code's objective measures of performance including sub-test results (e.g., whether it runs, whether it produces correct answers to unit tests etc.). Using these data, we will ask evaluators to judge the quality of the code using the same Likert scales as on the platform.

We aim to assess whether software developers evaluate pieces of code differently if they know the gender of the coder, controlling for the objective quality of the code. Our experiment uses a large set of de-identified code blocks written by a set of men and women on an online platform. This spans coders of different skill and problems of different levels of difficulty. For each code block, we will have access to the code's objective measures of performance including sub-test results (e.g., whether it runs, whether it produces correct answers to unit tests etc.). Using these data, we will ask evaluators to judge the quality of the code using the same Likert scales as on the platform.

Depending on the treatment condition, the evaluator will be aware of the gender or other basic information about the programmer who wrote the code (but will never be given identifying information). Using these evaluations, we will ask: (i) whether there are perceived differences in the quality of the code written by men and women; and (ii) how those perceived differences change when the evaluator is aware of the gender of the coder. This will let us test whether there are any unobservable dimensions of performance that are correlated with gender and driving the residual gender gaps that we see in our data. We will attempt to understand any such residual gaps by examining particular dimensions of performance as laid out in our pre-analysis plan.

2023-01-15

2023-02-15

Subjective ratings of code quality

Our primary outcome is evaluators’ subjective ratings of the code quality, whether it differs by the gender of the coder and by the treatment condition. For each block of code, respondents will be asked to rate problems on a scale from 1 to 4.

Evaluators’ prediction of the candidate’s score from the automated evaluation tool.

This is a continuous variable from [0,1]. A third outcome variable is evaluators’ prediction of the candidate hireability score. This is measured on a Likert-scale from 1 to 4, and allows us to draw a more direct link between our findings and hiring outcomes. Additionally, we will measure how much time respondents spend on each question to measure fatigue and inattention, and how this varies over time.

Our design relies on multiple observations per subject. Each participant will evaluate 4 code blocks.

Let N be the number of evaluators and P the number of problems by evaluator. As mentioned before, our sample of code blocks is stratified by gender and performance, such that P/2 code blocks are written by women, among which P/4 are high-scoring code blocks according to the platform objective device. Each evaluator i is assigned a set of P problems in a random order. We use a within-subject design. We define Rj = 0 for a blind problem j (if the gender of the coder is not revealed), Rj = 1 for a non-blind problem j (if the gender of the coder is revealed). For each evaluator i, the gender of the coder will be revealed for half of the problems. To account for potential priming effect, we plan to randomize whether the gender of the coder is revealed in the first or in the second half of the study.

Randomization will be done by a computer.

We use a within-subject design: half of the codes seen by an evaluator will be in the treated group, half in the control group. The order in which the treated half of the codes is seen will be randomized. The order of the codes within each condition will be randomized.

No

400 evaluators.

1600 observations.

800 treatment code blocks, 800 non-non-treated.

Pre-analysis plan