A Field Experiment on the Future of Software Development

The experiment runs on a single day at the partner firm's sites in India. Participants work on two blocks of seven programming tasks each, with 3.5 hours per block and a 1.5-hour break between the blocks. The tasks implement features in an internally developed document-management application whose source code has never been published online, so the tasks are novel both for the participants and for the AI models. Participants write code in Visual Studio Code and submit on an experimental platform as often as they wish within the block. After each submission the platform runs the task's unit tests on the submitted code and returns only the number of tests passed. The test code itself is never shown.

The first intervention is AI access, randomised between participants in three arms. In the No AI arm, participants work without any AI assistance. In the Weaker AI arm, participants receive GitHub Copilot restricted to Claude Haiku. In the Stronger AI arm, participants receive GitHub Copilot with Claude Haiku, Claude Sonnet, and Claude Opus. Participants in the two AI arms receive a 100-dollar AI credit budget per block, refreshed during the break.

The second intervention is the format of the task requirements, varied within participants across the two blocks. One block presents each task as a formal assignment, a structured specification that states the goal, the scope, and every rule and parameter explicitly. The other block presents each task as the transcript of an interview in which a non-technical product stakeholder describes what they need in informal language. Both versions carry the information needed to pass every unit test, so the formats differ in how the requirements are presented and the interview format requires the participant to extract and formalise them before coding. Half the participants receive the formal block first and half receive the interview block first.

Three features of the intervention are not communicated to participants and should stay hidden until the trial completes.

First, participants are not told that the second block presents the task requirements in the other format. Each participant discovers the format of a block only when they open it.

Second, part of the incentive system is undisclosed. Participants are told that bonuses for engagement and performance exist on top of the attendance compensation, but the eligibility rules for the engagement draw are not communicated: eligibility requires presence for the full session (or solving all tasks of a block), at least one valid code submission in each block, and recorded work activity in at least half of each block's 30-minute intervals.

Third, participants are not informed of the research hypotheses or of the treatment-arm structure. The firm removed every participant's existing access to AI tools without announcing it in advance, and each participant learns only their own condition at first login.

2026-08-07

2026-08-08

Block score: the share of rubric points earned across the block's seven tasks, a value between 0 and 1, with one observation per participant per block.

Each task is scored as the fraction of its rubric points earned. A script reports a pass or fail for every unit test, each test carries a pre-registered point weight from the firm's scoring rubric (two points for core-functionality tests, one point for edge-case tests), and the task score is the points earned divided by the task's total points. The block score is the mean of the seven task scores, so every task carries the same weight regardless of how many unit tests it has. A task with no submission in the block scores zero. If the per-test export fails validation before analysis, the task score falls back, by pre-registered rule, to the fraction of the task's unit tests that pass.

See pre-registration PDF

See pre-registration PDF

Population and stratification.

The partner firm registered its employees for the study through a recruitment grid crossing role type with tenure. Role types follow the firm's records: Experts (employees on a leadership and technical-authority track), Developers (employees hired to deliver code-based assignments), and Non-coders (employees in adjacent functions such as sales and marketing, typically without formal coding skills). Tenure has three levels: junior (under 3 years), mid (3 to 8 years), and senior (8 years or more). The randomised population is the 6,806 registrants who completed at least one pre-experimental assessment.

Treatments.
Two treatments are randomised independently. AI access is a between-participant factor with three arms (No AI, Weaker AI, Stronger AI, described in the Intervention field). Requirement format is a within-participant crossover: each participant completes one block with formal assignments and one block with stakeholder-interview transcripts, and the order is randomised with probability one half. Experts and Developers are assigned to No AI with probability 1/5 and to each AI arm with probability 2/5. Non-coders are assigned only to the two AI arms with probability 1/2 each, because asking participants without coding skills to complete programming tasks without AI would be unfair and demotivating; their No AI performance is set to zero. All participants receive the same two blocks in the same order (the block content is fixed, the format is not), the seven tasks within a block appear in a randomly shuffled order per participant, and participants choose freely which task to attempt and when.

Tasks.
The 14 experimental tasks were selected from 20 candidates developed with the firm's engineering team, each with unit tests and a scoring rubric validated by the firm. Tasks that the strongest available model solved at the first attempt from the pasted formal assignment were removed, and the remaining tasks were split into two blocks balanced on a machine difficulty index (from an LLM-alone baseline) and a human difficulty index (from expert rankings).

Pre-experimental measurement.
Before the experiment day, every participant completes a battery of assessments: a registration survey, a professional-experience survey, a coding-skills survey, a timed 15-item coding assessment, an LLM-expertise survey, an LLM-driven career interview, and a cognitive and personality battery (Raven's Progressive Matrices, Digit Span, Cognitive Flexibility Inventory, Ten-Item Personality Inventory, General Risk Propensity Scale, and a delay-discounting task). Six confirmatory expertise measures come from this battery: coding knowledge (the timed coding assessment score), career experience (total professional experience in years), on-the-job coding time (the share of the working week spent on the work the tasks exercise), AI expertise (a five-component index of GenAI knowledge and use), fluid intelligence (Raven's score), and working memory (total Digit Span).

Hypotheses.
We test nine confirmatory hypotheses.
- H1: the effect of AI on task performance is positive, and larger for Stronger than for Weaker AI.
- H2: the effect of AI is smaller when requirements arrive as a stakeholder interview rather than a formal assignment, and more so for Stronger AI.
- H3 to H8: the effect of AI is greater when coding knowledge (H3), career experience (H4), on-the-job coding time (H5), AI expertise (H6), fluid intelligence (H7), or working memory (H8) is higher, in each case more so for Stronger AI.
- H9: AI access transforms the relative value of knowledge and cognitive traits, so the effect of AI grows more with an index of cognitive traits (fluid intelligence and working memory) than with an index of knowledge (coding knowledge, on-the-job coding time, and career experience).

Estimation.
The hypotheses are tested in one ordinary-least-squares regression per block. The block score is regressed on the two AI-arm indicators, the format indicator, the six z-scored expertise measures, and the full set of their interactions. H9 uses a companion regression per block that replaces the six measures with three range-normalised indexes (cognitive traits, knowledge, AI expertise). An LLM-alone baseline, in which the same models attempt every task under both formats with three prompt scripts and three repetitions, provides each task's machine difficulty index and the benchmark for value added.

Exploratory analysis.
Heterogeneous effects of both treatments are estimated with a post-selection Lasso (with selective inference) and an honest causal forest over the pre-treatment covariates, with best-linear-projection, group-average-effect, and RATE summaries. Primary analyses include every participant who was assigned a condition; pre-specified robustness subsets are participants with at least one valid submission, participants meeting the engagement threshold, and each participant's first block only.

Randomisation done by computer by the research team, on an anonymised list of registered employees supplied by the partner firm, where tenure and role it's disclosed. AI access is assigned by permuted-block randomisation within role-type by tenure strata, with the arm probabilities stated in the design field. Format order is assigned with probability one half.

The individual participant, for both treatments. AI access is randomised between participants. Requirement format is a within-participant crossover, so the randomised unit for the second treatment is the order in which one participant receives the two formats.

No

Randomisation is at the individual level; there are no clusters. 6,806 individuals were randomised.

6,806 randomised employees, of whom approximately 5,000 are expected to attend the experiment day. Each attending participant contributes two block-level observations, one per requirement format, for up to 13,612 participant-block observations.

No AI: 849 participants (424 formal-assignment block first, 425 stakeholder-interview block first). Weaker AI: 2,977 participants (1,489 formal first, 1,488 interview first). Stronger AI: 2,980 participants (1,488 formal first, 1,492 interview first). By role type, the No AI / Weaker AI / Stronger AI split is 307 / 614 / 616 for Experts, 542 / 1,084 / 1,087 for Developers, and 0 / 1,279 / 1,277 for Non-coders, because Non-coders are assigned only to the AI arms. Every participant receives both requirement formats, one per block.