Can AI help teachers? Experimental Evidence on the effects of Generative AI-use and AI training for teachers' work

Study participants receive an introduction into the use of LLMs. While some components of the introduction are rather general, it is particularly tailored for the preparation of teaching material in adult education. The training is thus supposed to support participants in using LLMs to enhance the quality of teaching material and reduce the time needed for its preparation.

Study participants receive an introduction into the use of LLMs. While some components of the introduction are rather general, it is particularly tailored for the preparation of teaching material in adult education. The training is thus supposed to support participants in using LLMs to enhance the quality of teaching material and reduce the time needed for its preparation.
The training consists of three modules. The first module focusses on the general functioning of LLMs. The second part focusses on effective communication with the LLMs. We provide different examples on how a basic prompt can be adapted to change the answer of the LLM in the desired way. We point out that providing context information is crucial to receive the desired output. Moreover, we point to special prompting techniques such as including relevant sources into the prompt (e.g. pasting the whole text into the prompt if one wants to create questions about the text). Modules 1 and 2 mainly consist of learning videos provided by the KI Campus, a network of professionals dedicated to enhance the skilled use of KI in practice, especially teaching. After the videos, we provide key learnings that summarize the most important aspects. In the third part, we provide exercises that participants are supposed to solve using ChatGPT (embedded into the experimental environment by means of an API). After participants have finished the exercises, we provide videos with solutions for possible prompts that have led to the desired output by the LLM. The study is conduced fully online with the software labvanced.

2024-12-05

2025-01-31

Overall performance in completing the tasks

Performance in the test is measured by the quality of the output, rated by external reviewers, as well as the time spent to complete the tasks. As primary outcome, we limit ourselves to the aggregate performance over all four tasks.

Disaggregate analyses for separate tasks
Political attitudes

For exploratory purposes, we analyse the effect of the treatment on performance at the four tasks separately. Moreover, we look at the attitudes towards AI as measured by two items asking for expected (negative) employment and (positive) productivity effects in the industrial sector the participants are working in.

The experiment is conducted as online experiment with delayed control group design.

The whole intervention consists of two parts: the tasks to be completed and the intervention itself (the LLM training). We split participants randomly in three groups. The first group is asked to complete the tasks prior to receiving the training. Moreover, this group does not get access to an LLM but is asked to complete the tasks without using LLMs. The second group is also asked to complete the tasks prior to the training but gets access to the most recent version of ChatGPT embedded in the experimental environment. The third group receives the training first and is then asked to complete the tasks.

These tasks are 1) preparing a quiz on the political system of Germany for the first hour of an integration course for migrants, 2) writing an e-mail to participants of an integration course prior to the first session to welcome them in the course, 3) extending a textbook exercise from a language course (the original exercises consists of five sentences where participants have to decide whether a verb has to be spelled with small or capital letter; participants are asked to create ten additional analogue sentences) and 4) creating questions to a text on majority vote and proportional representation. For all the four tasks, we record the time used to complete the task. Moreover, we ask external persons to rate the quality of the answers. We will provide a set of relevant criteria as basis for the rating.

Randomization is conducted by the software labvanced when participants enter the intervention (simple randomization)

Individuals

No

No clustering, invitation and randomization is conducted at the individual level

We invite 800 persons to participate in the study. We expect 160 participants in total.

We expect to have between 50 and 60 persons per treatment arm