Zooming to Class?: Evidence from a Randomized Control Trial on the Effects of Online Learning on College Student Academic Achievement

Last registered on December 22, 2020

Pre-Trial

Trial Information

General Information

Title
Zooming to Class?: Evidence from a Randomized Control Trial on the Effects of Online Learning on College Student Academic Achievement
RCT ID
AEARCTR-0006969
Initial registration date
December 22, 2020

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
December 22, 2020, 3:22 PM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Primary Investigator

Affiliation
United States Military Academy

Other Primary Investigator(s)

Additional Trial Information

Status
Completed
Start date
2020-08-24
End date
2020-12-11
Secondary IDs
Abstract
The COVID-19 pandemic caused colleges and universities to offer many courses online. The United States Military Academy at West Point was no exception and each department was asked to find a way to teach classes at 50 percent capacity. Per institutional and Army policy, West Point already randomly assigns cadets to course order, class hour, instructor, and class section. The Economics Department decided have each instructor teach half of their course load online and the other half face to face. Given West Point's previous policy on random assignment to class section, cadets were randomly assigned to either an online or face to face class. Instructors used a uniform curriculum and all graded events were the same across the modality. This experimental setting will allow us to control for instructor style and quality while assessing the efficacy of online learning.
External Link(s)

Registration Citation

Citation
Kofoed, Michael. 2020. "Zooming to Class?: Evidence from a Randomized Control Trial on the Effects of Online Learning on College Student Academic Achievement." AEA RCT Registry. December 22. https://doi.org/10.1257/rct.6969-1.0
Experimental Details

Interventions

Intervention(s)
Intervention Start Date
2020-08-24
Intervention End Date
2020-12-11

Primary Outcomes

Primary Outcomes (end points)
Our primary outcomes are student grades for each assignment or exam. We will also include questions on an optional course evaluation that will allow us to understand the mechanisms behind any results including questions about concentration in class, connectedness to instructor and/or peers, etc.
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
In this paper, we leveraged unique institutional characteristics at the United States Military Academy at West Point to implement a randomized control trial for a required Principles of Economics course to estimate differences in student outcomes between online or in-person instruction. In a non-COVID environment, West Point cadets (along with their peers at the Air Force and Naval Academies) have little control over their academic schedules and peer networks. For example, they do not choose which semester they are taking a specific class and are randomly assigned to academic instructors, class hours, roommates, social networks called companies, final exam periods, and military mentors.

While West Point continued to sort cadets in this matter during the COVID-19 pandemic, the Academy's administration requested each department to create a plan to use physical classroom space at fifty percent capacity. Departments were free, however, to innovate a solution to carry out this requirement. Some departments, for example, moved to a hybrid model where half of the enrolled cadets meet in person for one class meeting and the other half met online and switched for the next class time. To address this concern, the Economics Department received permission to have each instructor assigned to the introductory Principles of Economics to teach half of our sections online and the other half face to face. This course load split allows us to control for instructor quality and experience in both modalities. Principles of Economics is also a graduation requirement for all cadets, so there are no concerns about selection into the course. We randomized 550 cadets across over twelve instructors for thirty-eight class sections. In our experiment, all curriculum, graded events, homework assignments, and exams were identical across online and face to face. We also offered both online and in-person class sections for each class-hour that the course was available and we randomly assigned cadets within the hour into each modality. Cadets randomly assigned to an online class will see their grades increased by any estimated treatment effect to ensure they are not harmed by the experiment.
Experimental Design Details
Randomization Method
Randomization (before/during/after) COVID is done by the Office of the Dean of the Academic Board (Provost) using a computer algorithm.
Randomization Unit
Classroom
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
38 class sections
Sample size: planned number of observations
550 students over 12 instructors
Sample size (or number of clusters) by treatment arms
214 in a face to face class with 337 online.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
IRB

Institutional Review Boards (IRBs)

IRB Name
United States Military Academy Institutional Review Board
IRB Approval Date
2020-08-19
IRB Approval Number
21-014

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials