x

Please fill out this short user survey of only 3 questions in order to help us improve the site. We appreciate your feedback!
Zooming to Class?: Evidence from a Randomized Control Trial on the Effects of Online Learning on College Student Academic Achievement
Last registered on December 22, 2020

Pre-Trial

Trial Information
General Information
Title
Zooming to Class?: Evidence from a Randomized Control Trial on the Effects of Online Learning on College Student Academic Achievement
RCT ID
AEARCTR-0006969
Initial registration date
December 22, 2020
Last updated
December 22, 2020 3:22 PM EST
Location(s)
Primary Investigator
Affiliation
United States Military Academy
Other Primary Investigator(s)
Additional Trial Information
Status
Completed
Start date
2020-08-24
End date
2020-12-11
Secondary IDs
Abstract
The COVID-19 pandemic caused colleges and universities to offer many courses online. The United States Military Academy at West Point was no exception and each department was asked to find a way to teach classes at 50 percent capacity. Per institutional and Army policy, West Point already randomly assigns cadets to course order, class hour, instructor, and class section. The Economics Department decided have each instructor teach half of their course load online and the other half face to face. Given West Point's previous policy on random assignment to class section, cadets were randomly assigned to either an online or face to face class. Instructors used a uniform curriculum and all graded events were the same across the modality. This experimental setting will allow us to control for instructor style and quality while assessing the efficacy of online learning.
External Link(s)
Registration Citation
Citation
Kofoed, Michael. 2020. "Zooming to Class?: Evidence from a Randomized Control Trial on the Effects of Online Learning on College Student Academic Achievement." AEA RCT Registry. December 22. https://doi.org/10.1257/rct.6969-1.0.
Experimental Details
Interventions
Intervention(s)
Intervention Start Date
2020-08-24
Intervention End Date
2020-12-11
Primary Outcomes
Primary Outcomes (end points)
Our primary outcomes are student grades for each assignment or exam. We will also include questions on an optional course evaluation that will allow us to understand the mechanisms behind any results including questions about concentration in class, connectedness to instructor and/or peers, etc.
Primary Outcomes (explanation)
Secondary Outcomes
Secondary Outcomes (end points)
Secondary Outcomes (explanation)
Experimental Design
Experimental Design
In this paper, we leveraged unique institutional characteristics at the United States Military Academy at West Point to implement a randomized control trial for a required Principles of Economics course to estimate differences in student outcomes between online or in-person instruction. In a non-COVID environment, West Point cadets (along with their peers at the Air Force and Naval Academies) have little control over their academic schedules and peer networks. For example, they do not choose which semester they are taking a specific class and are randomly assigned to academic instructors, class hours, roommates, social networks called companies, final exam periods, and military mentors.

While West Point continued to sort cadets in this matter during the COVID-19 pandemic, the Academy's administration requested each department to create a plan to use physical classroom space at fifty percent capacity. Departments were free, however, to innovate a solution to carry out this requirement. Some departments, for example, moved to a hybrid model where half of the enrolled cadets meet in person for one class meeting and the other half met online and switched for the next class time. To address this concern, the Economics Department received permission to have each instructor assigned to the introductory Principles of Economics to teach half of our sections online and the other half face to face. This course load split allows us to control for instructor quality and experience in both modalities. Principles of Economics is also a graduation requirement for all cadets, so there are no concerns about selection into the course. We randomized 550 cadets across over twelve instructors for thirty-eight class sections. In our experiment, all curriculum, graded events, homework assignments, and exams were identical across online and face to face. We also offered both online and in-person class sections for each class-hour that the course was available and we randomly assigned cadets within the hour into each modality. Cadets randomly assigned to an online class will see their grades increased by any estimated treatment effect to ensure they are not harmed by the experiment.
Experimental Design Details
Randomization Method
Randomization (before/during/after) COVID is done by the Office of the Dean of the Academic Board (Provost) using a computer algorithm.
Randomization Unit
Classroom
Was the treatment clustered?
Yes
Experiment Characteristics
Sample size: planned number of clusters
38 class sections
Sample size: planned number of observations
550 students over 12 instructors
Sample size (or number of clusters) by treatment arms
214 in a face to face class with 337 online.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
IRB
INSTITUTIONAL REVIEW BOARDS (IRBs)
IRB Name
United States Military Academy Institutional Review Board
IRB Approval Date
2020-08-19
IRB Approval Number
21-014
Post-Trial
Post Trial Information
Study Withdrawal
Intervention
Is the intervention completed?
No
Is data collection complete?
Data Publication
Data Publication
Is public data available?
No
Program Files
Program Files
Reports, Papers & Other Materials
Relevant Paper(s)
REPORTS & OTHER MATERIALS