Assessing Generative AI value in a public sector context: evidence from a field experiment

Last registered on April 26, 2024
View Trial History

Pre-Trial

Trial Information

General Information

Title
Assessing Generative AI value in a public sector context: evidence from a field experiment
RCT ID
AEARCTR-0013347
Initial registration date
April 24, 2024

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
April 26, 2024, 12:38 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Primary Investigator

Name
Trevor Fitzpatrick
Affiliation
Central Bank of Ireland
Contact Primary Investigator

Other Primary Investigator(s)

PI Name
David Walsh
PI Affiliation
Central Bank of Ireland
Contact Investigator
PI Name
Seamus Kelly
PI Affiliation
Central Bank of Ireland
Contact Investigator
PI Name
Patrick Carey
PI Affiliation
Central Bank of Ireland
Contact Investigator
PI Name
Ruairi Nugent
PI Affiliation
Central Bank of Ireland
Contact Investigator
PI Name
Tom Cleary
PI Affiliation
Central Bank of Ireland
Contact Investigator

Additional Trial Information

Status
In development
Start date
2024-04-24
End date
2024-07-10
Keywords
Firms & Productivity, Other
Additional Keywords
Generative AI, Artificial Intelligence, causal inference
JEL code(s)
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
Generative AI economic benefits in aggregate are often cited. However, the productivity or efficiency improvements within organisations is an emerging topic. Recent research focused on knowledge work suggest productivity or efficiency improvements may be large, finding significant effects for certain types of tasks, with more benefits to less experienced workers. Evaluating the value/benefits and costs of GenAI in knowledge-driven organisations within a public sector may not be as clear-cut in a for-profit context. To provide some evidence, our study will test the impact of GenAI on two knowledge-work tasks among staff in the organisation.
External Link(s)

Registration Citation

Citation
Carey, Patrick et al. 2024. "Assessing Generative AI value in a public sector context: evidence from a field experiment ." AEA RCT Registry. April 26. https://doi.org/10.1257/rct.13347-1.0
Sponsors & Partners
Experimental Details

Interventions

Intervention(s)
The potential productivity or efficiency improvements of Generative AI within organisations is an emerging research topic, including the potential effects for knowledge workers. Recent research (Dell'Acqua et al., 2023) focused on knowledge work suggest productivity or efficiency improvements may be large, finding significant effects for certain types of tasks, with more benefits to less experienced staff. To provide initial evidence, our study will test the impact of GenAI on two representative knowledge-work tasks among staff in the organisation. The first task is data related, and the second task is document based. These differ from the types of tasks considered so far in the literature.
Intervention Start Date
2024-05-01
Intervention End Date
2024-05-10

Primary Outcomes

Primary Outcomes (end points)
The key outcome variable for each participant is quality of the response measured by the overall quality rating (graded by independent grader's according to a rubric) and objective measures (% of tasks completed correctly)
Primary Outcomes (explanation)
Task completion (% of tasks completed correctly)
Task Quality (a quality rating)

Secondary Outcomes

Secondary Outcomes (end points)
Task Time (duration in mins/seconds)
Secondary Outcomes (explanation)
The time taken to complete the overall task in minutes/seconds. This is self reported.

Experimental Design

Experimental Design
A two phase design, with a pre-assessment task to baseline individual subjects and two tasks within the main experiment. The control group receives task instructions, data or documents to work from. The treatment group receives task instructions, data/documents to work from and the assistance of an LLM focused on these types of tasks.
Experimental Design Details
Not available
Randomization Method
Done in office by computer
Randomization Unit
Subject
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
0
Sample size: planned number of observations
152
Sample size (or number of clusters) by treatment arms
38 for treatment (LLM use); 38 control (no LLM use) for each task
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Supporting Documents and Materials
IRB

Institutional Review Boards (IRBs)

IRB Name
IRB Approval Date
IRB Approval Number
Analysis Plan

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information