Can AI help teachers? Experimental Evidence on the effects of Generative AI-use and AI training for teachers' work

Last registered on December 10, 2024

Pre-Trial

Trial Information

General Information

Title
Can AI help teachers? Experimental Evidence on the effects of Generative AI-use and AI training for teachers' work
RCT ID
AEARCTR-0014952
Initial registration date
December 05, 2024

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
December 10, 2024, 11:24 AM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Region

Primary Investigator

Affiliation
DIE - Leibniz centre for Life-long Learning

Other Primary Investigator(s)

PI Affiliation
DIE - Leibniz Institute for Life-Long Learning

Additional Trial Information

Status
In development
Start date
2024-12-05
End date
2025-01-31
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
Artificial intelligence, in particular Large Language Models (LLMs), are commonly considered being a path-breaking innovation. While possible applications and rising research questions are manifold, one straightforward question of very general interest is whether LLMs can support humans in fulfilling job-related tasks, hereby improving the quality and quantity of the output. In this study, we will focus on the impact of LLM and LLM training on teachers in adult education. To this end, we design an experimental study in which we ask participants to complete four tasks typical for teaching preparation and communication with course participants. We vary both access to an assisting LLM as well as being trained in the use of LLMs before completing the tasks. This will provide insights about whether and to what extend adult education teachers can profit from the use of LLMs or being trained in their use to improve the quality of teaching material and/or increase the efficiency in teaching preparation.
External Link(s)

Registration Citation

Citation
Appel, Philipp and Lukas Fervers. 2024. "Can AI help teachers? Experimental Evidence on the effects of Generative AI-use and AI training for teachers' work." AEA RCT Registry. December 10. https://doi.org/10.1257/rct.14952-1.0
Experimental Details

Interventions

Intervention(s)
Study participants receive an introduction into the use of LLMs. While some components of the introduction are rather general, it is particularly tailored for the preparation of teaching material in adult education. The training is thus supposed to support participants in using LLMs to enhance the quality of teaching material and reduce the time needed for its preparation.
Intervention Start Date
2024-12-05
Intervention End Date
2025-01-31

Primary Outcomes

Primary Outcomes (end points)
Overall performance in completing the tasks
Primary Outcomes (explanation)
Performance in the test is measured by the quality of the output, rated by external reviewers, as well as the time spent to complete the tasks. As primary outcome, we limit ourselves to the aggregate performance over all four tasks.

Secondary Outcomes

Secondary Outcomes (end points)
Disaggregate analyses for separate tasks
Political attitudes
Secondary Outcomes (explanation)
For exploratory purposes, we analyse the effect of the treatment on performance at the four tasks separately. Moreover, we look at the attitudes towards AI as measured by two items asking for expected (negative) employment and (positive) productivity effects in the industrial sector the participants are working in.

Experimental Design

Experimental Design
The experiment is conducted as online experiment with delayed control group design.
Experimental Design Details
Not available
Randomization Method
Randomization is conducted by the software labvanced when participants enter the intervention (simple randomization)
Randomization Unit
Individuals
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
No clustering, invitation and randomization is conducted at the individual level
Sample size: planned number of observations
We invite 800 persons to participate in the study. We expect 160 participants in total.
Sample size (or number of clusters) by treatment arms
We expect to have between 50 and 60 persons per treatment arm
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Supporting Documents and Materials

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
IRB

Institutional Review Boards (IRBs)

IRB Name
Local Ethics Committee of the German Institute for Adult Education
IRB Approval Date
2024-08-26
IRB Approval Number
DIE-LEK 2024-007
Analysis Plan

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information