The Effects of Generative AI on High Skilled Work: Evidence from Three Field Experiments with Software Developers

Last registered on October 07, 2024

View Trial History

Pre-Trial

Trial Information

General Information

Title

The Effects of Generative AI on High Skilled Work: Evidence from Three Field Experiments with Software Developers

RCT ID

AEARCTR-0014530

Initial registration date

October 07, 2024

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published

October 07, 2024, 7:24 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Country

United States of America

Region

Primary Investigator

Name

sida peng

Affiliation

Microsoft

Contact Primary Investigator

Other Primary Investigator(s)

Additional Trial Information

Status

Completed

Start date

2022-01-01

End date

2024-05-05

Keywords

Firms & Productivity

Additional Keywords

JEL code(s)

J24

Secondary IDs

Prior work

This trial does not extend or rely on any prior RCTs.

Abstract

This study evaluates the impact of generative AI on software developer productivity by
analyzing data from three randomized controlled trials conducted at Microsoft, Accenture,
and an anonymous Fortune 100 electronics manufacturing company. These field
experiments, which were run by the companies as part of their ordinary course of business,
provided a randomly selected subset of developers with access to GitHub Copilot, an AIbased
coding assistant that suggests intelligent code completions.

External Link(s)

Registration Citation

Citation

peng, sida. 2024. "The Effects of Generative AI on High Skilled Work: Evidence from Three Field Experiments with Software Developers." AEA RCT Registry. October 07. https://doi.org/10.1257/rct.14530-1.0

Sponsors & Partners

Partner

Name

Microsoft

Type

private_company

URL

https://microsoft.com

Experimental Details

Interventions

Intervention(s)

The study granted a randomly selected subset of software developers access to GitHub
Copilot, an AI-based coding assistant that suggests intelligent code completions.

Intervention (Hidden)

Intervention Start Date

2022-09-01

Intervention End Date

2023-05-01

Primary Outcomes

Primary Outcomes (end points)

# Pull Requests, # Commits, # Builds, Build Success Rate, # Accepted Suggestions,
Fraction of Developers Using Copilot (i.e., Adoption)

Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)

Secondary Outcomes (explanation)

Experimental Design

Initially, a random subset of developers is granted access to GitHub Copilot. During the
intervention period, developers not in our treatment group cannot use Copilot. After the
intervention period, all developers are able to use Copilot, but there may still be differential
adoption across treatment and control groups, and we also utilize this variation.

Experimental Design Details

Randomization Method

computer-based randomization

Randomization Unit

For 50% of teams at Microsoft, the treatment is randomized by team; for the remaining 50%,
it is assigned randomized by individual. At Accenture, the treatment is randomized
individually. At anonymous company, the treatment is clustered by team.

Was the treatment clustered?

Yes

Experiment Characteristics

Sample size: planned number of clusters

1438

Sample size: planned number of observations

4869

Sample size (or number of clusters) by treatment arms

Microsoft: 70 teams assigned to treat, 79 assigned to control; for remaining 152 teams,
randomization is individual with 755 individuals treated, 766 individuals control.
Accenture: 193 developers treated, 123 developers in control
Anon. Company: roughly equally split 432 teams into 6 rollout weeks (each week, another
~16% of teams get access)

Minimum detectable effect size for main outcomes (accounting for sample design and clustering)

Supporting Documents and Materials

IRB