A Field Experiment on Adaptive Learning Applying a Machine Learning Algorithm

Last registered on May 26, 2021
View Trial History

Pre-Trial

Trial Information

General Information

Title
A Field Experiment on Adaptive Learning Applying a Machine Learning Algorithm
RCT ID
AEARCTR-0007637
Initial registration date
May 26, 2021
Last updated
May 26, 2021, 10:28 AM EDT

Locations

Country
Germany
Region

Primary Investigator

Name
Isabell Zipperle
Affiliation
Johannes Gutenberg University Mainz
Contact Primary Investigator

Other Primary Investigator(s)

PI Name
Tim Klausmann
PI Affiliation
Johannes Gutenberg University Mainz
Contact Investigator
PI Name
Marius Köppel
PI Affiliation
Johannes Gutenberg University Mainz
Contact Investigator
PI Name
Daniel Schunk
PI Affiliation
Johannes Gutenberg University Mainz
Contact Investigator

Additional Trial Information

Status
In development
Start date
2018-01-22
End date
2022-12-31
Keywords
Education
Additional Keywords
Education, Field Experiment, Digitalization, Machine Learning, Learning Behavior, Adaptive Learning, Dynamic Difficulty Adjustment
JEL code(s)
C69, C93, I20, I21
Secondary IDs
Abstract
It remains an open question how to adapt and individualize learning contents. To tackle this in a digital learning context, we developed an algorithm based on a convolutional neural network that assigns tasks to the learners. Our application is a large online learning platform in which we run a randomized controlled trial. Participants are randomized into three groups: two treatment groups – a group-based adaptive treatment group and an individualized adaptive treatment group – and one control group. We analyze the difference between the three groups with respect to effort learners provide within the platform, their performance within the platform, formative assessments within the platform and their final high-stake standardized exams (summative assessment).
External Link(s)

Registration Citation

Citation
Klausmann, Tim et al. 2021. "A Field Experiment on Adaptive Learning Applying a Machine Learning Algorithm." AEA RCT Registry. May 26. https://doi.org/10.1257/rct.7637-1.0
Sponsors & Partners

There are documents in this trial unavailable to the public. Use the button below to request access to this information.

Request Information
Experimental Details

Interventions

Intervention(s)
There is a control group, a group-based adaptive treatment group, and an individualized adaptive treatment group. The control group receives tasks randomly drawn from the pool of tasks available in the learning platform. Both treatment groups are affected by our “intervention”. Tasks are assigned to learners based on pre-tested distributions that optimize task difficulty and learning effort, which is the number of future tasks that learners will complete in the platform. A machine learning algorithm predicts task difficulty and future effort. The groups differ in the degree of individualization of assigned tasks. The group-based adaptive treatment group receives tasks that are adapted to the mean learner in the platform. The individualized adaptive treatment group receives tasks that are adapted to each individual learner.
Intervention Start Date
2021-05-26
Intervention End Date
2021-12-31

Primary Outcomes

Primary Outcomes (end points)
Number of attempted tasks
Primary Outcomes (explanation)
Cumulative number of tasks attempted on the paltform throughout the intervention period
Measure for effort

Secondary Outcomes

Secondary Outcomes (end points)
Test score in formative assessments (mock exams) on the platform
Test score in summative assessments (standardized high-stake final exams)
Secondary Outcomes (explanation)
Measure for performance

Experimental Design

Experimental Design
The setting of our experiment is a college-level online learning platform that students use outside the classroom (for more information of the platform see Sponsors & Partner section). Learning in the platform is unmonitored by their teachers. Originally, the platform consists of learning videos and tasks that are matched to the videos and that follow the purpose of deepening the understanding of the video contents. We added a section to the platform that combines learning materials to quizzes of ten tasks. Our intervention algorithm assigns tasks to the learners whenever they choose to take a quiz.
Experimental Design Details
Not available
Randomization Method
Computer-based randomization by script on a server of the cooperation partner
Randomization Unit
Individual
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
10,000
Sample size: planned number of observations
10,000
Sample size (or number of clusters) by treatment arms
3,300 group-based adaptive treatment group
3,300 individualized adaptive treatment group
3,300 control group
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Based on a previous study in the same platform, we estimate that we have 80 percent power at an effect size of 0.057 standard deviations (SD). Equivalently, we reach 99 percent power at an effect size of 0.095 SD. As inputs in our power calculation, we chose alpha = 0.05, N = 10,000 and R²=0.005.
Supporting Documents and Materials
IRB

Institutional Review Boards (IRBs)

IRB Name
Joint Ethics Committee of the Faculty of Economics and Business Administration of Goethe University Frankfurt and the Gutenberg School of Management & Economics of the Faculty of Law, Management and Economics of Johannes Gutenberg University Mainz
IRB Approval Date
2021-05-25
IRB Approval Number
N/A
Analysis Plan