What Drives Different Assessments of Performance on Coding Tasks?

We aim to assess how software developers evaluate pieces of code written by computer programmers. Our experiment uses a large set of de-identified code blocks from an online coding platform, which span coders of different skill and problems of different levels of difficulty. For each code block, we will have access to the code's objective measures of performance including sub-test results (e.g., whether it runs, whether it produces correct answers to unit tests etc.). Using these data, we will ask evaluators to judge the quality of the code using the same Likert scales as on the platform.

2023-01-15

2023-02-15

Subjective ratings of code quality

Our primary outcome is evaluators’ subjective ratings of the code quality, whether it differs by the gender of the coder and by the treatment condition. For each block of code, respondents will be asked to rate problems on a scale from 1 to 4.

Evaluators’ prediction of the candidate’s score from the automated evaluation tool.

This is a continuous variable from [0,1]. A third outcome variable is evaluators’ prediction of the candidate hireability score. This is measured on a Likert-scale from 1 to 4, and allows us to draw a more direct link between our findings and hiring outcomes. Additionally, we will measure how much time respondents spend on each question to measure fatigue and inattention, and how this varies over time.

Our design relies on multiple observations per subject. Each participant will evaluate 4 code blocks.

Not available

Randomization will be done by a computer.

We use a within-subject design: half of the codes seen by an evaluator will be in the treated group, half in the control group. The order in which the treated half of the codes is seen will be randomized. The order of the codes within each condition will be randomized.

No

400 evaluators.

1600 observations.

800 treatment code blocks, 800 non-non-treated.