The Impact of Online Lectures on Student Learning

In the exceptional circumstances prompted by COVID-19, universities around the world have massively increased the number of courses taught online. Even before 2020, the digitization of higher education began to transform the academic experience and learning process for university students. However, very limited research exists on how students are impacted by the shift to online lectures.
In this paper, we investigate how the introduction of online lectures affects student performance and university experience. We study a European research university that was forced to introduce online lectures as part of social distancing measures in 2020. In this setting, we assign students to a rotating attendance schedule that determines whether students attend live lectures or watch the same lectures online. Assignment to live or online lectures is determined by a random number – the last digit of the student identification number.
We aim to answer the following research questions:

1. Do students perform better on exam questions when the content of the question was covered in an online lecture compared to a traditional in-person lecture? (Exam performance)
2. Do online lectures affect subsequent attendance, course dropout, course passing, and student grades? (Academic trajectory)
3. Do online lectures affect students’ social interactions, satisfaction with different course features, study habits, and overall learning experience? (Subjective outcomes)
4. Does assignment to online attendance affect study dropout, completion of the first year, elective course choices, and major switching? (Longer run outcomes)

2020-09-14

2020-12-24

- Exam question answered correctly
- Subsequent lecture attendance
- Course dropout
- Course passing
- Student grades

The primary outcome of our exam question analysis is whether a student answered a given multiple choice question correctly. We link each exam question to lecture content to determine whether a given student-course-question observation is treated. This will allow us to directly estimate the impact of online lecture assignment on the probability of successfully answering a specific exam question.
The primary outcomes for our student-course level analysis are subsequent lecture attendance, course dropout, course passing, and student grades. These outcomes capture students’ academic trajectory. Course dropout and passing are binary outcomes, indicating if a student remained in the course or passed the course. The overall course grade is a continuous measure.
All primary outcomes will be measured through administrative data.

- Student subjective outcomes
- Study dropout
- Completion of first year
- Elective course choices
- Major switching

Secondary outcomes are student subjective outcomes as well as longer run educational outcomes. Student subjective outcomes are reported social interactions, satisfaction with different course features, study habits, and their overall learning experience. These outcomes will be measured through an endline survey.
Longer run educational outcomes will be study dropout, completion of the first year, elective course choices, and major switching. These outcomes will be measured through administrative data – if we receive the necessary data access.

Our field experiment takes place at a European research university. All first-year bachelor and master students are randomly assigned into five attendance groups. The assignment is based on a random number – the last digit of the student ID number. Based on a rotating block schedule, each attendance group is permitted to attend some lectures in person, and others online via Zoom. More specifically, each attendance group is allowed to come to the university two days per week . The randomization into attendance groups implies that, for any given lecture, some students receive the content in person and others receive it via an online lecture. Our within-subject design means that a given student attends some lectures online and others in person. Therefore, our experimental design creates exogenous variation in whether a given lecture was attended online and in the share of lectures a student attended online.
By linking lecture content to exam questions, our design allows us to compare student performance based on whether they attended a lecture online or in person. In other words, we will evaluate the impact of online lecture assignment on student performance through exam questions covering material taught in the corresponding lecture.

All first-year students are randomly assigned into five attendance groups based on the last digit of their student ID number. Each week, each group can come to university for two full days, according to a rotating block schedule. This randomization implies that for any given lecture, some students receive the content in person and others receive it via an online lecture.

Randomization into live and online lectures takes place within students. A given student will attend some lectures live and some online.

No

About 1000 students.

We expect to observe at least 1,000 students. For each of these students we expect to observe at least 50 responses to different multiple choice exam questions resulting in an overall sample of about 50,000 observations.

We expect a sample size of 50,000 exam questions. So about 10,000 questions in the control group and 40,000 questions in the treated (online) group.

We expect to observe at least 1,000 students. For each of these students we expect to observe at least 50 responses to different multiple choice exam questions resulting in an overall sample of about 50,000 observations. A simple power calculation suggests that we have sufficient statistical power to detect fairly small effect sizes. We assume a sample size of 50,000 exam questions, power of .8, and treatment arm that is 80% of the entire sample. For the binary outcome, we assume a mean of 0.7 correct answers and a standard deviation of 0.45. We conduct a two-sample means test to calculate the minimum detectable effects (MDEs) of a treatment effect significant at the five percent level. Here, δ refers to the difference between the in-person group mean and the online group mean for a given exam question. With a power of .8 we can detect differences between the treatment and control group that are larger than 0.0141. This corresponds to a minimum detectable treatment effect of 0.031 standard deviations.

Pre-Analysis Plan