Impact of Different Online Learning Modes on Student Performance and Engagement

Students of a Coursera Contracts course will be randomly assigned to one of the five groups. In each group, the substantive content will remain the same – same words spoken and same words shown in powerpoint slides and photos, but the groups will differ with the visual image and voice of the professor. Here is a description of the specific groups:
• Control group: Students in this group will learn the course material by being shown the preexisting videos with the professor delivering the material throughout the course
• Treatment group 1: Students in this group will learn the course by being shown videos in which the professor is replaced by a black male avatar instructor.
• Treatment group 2: Students in this group will learn the course by being shown videos in which the professor is replaced by a black female avatar instructor.
• Treatment group 3: Students in this group will learn the course by being shown videos in which the professor is replaced by a white male avatar instructor.
• Treatment group 4: Students in this group will learn the course by being shown videos in which the professor is replaced by a white female avatar instructor.

The following list presents specific questions of interest along with the appropriate statistical approaches to studying them:
1. Do students perform better when they are taught by avatar instructors than when they are taught by the professor?

Method 1: OLS regression
• Dependent variable: students’ overall grade
• Independent variables:
- Whether they were taught by an avatar instructor or by the real professor
- Number of lessons passed
- Student characteristics (age, race, gender, level of education)

Method 2: OLS regression (This method allows us to see a more detailed effect of each of the treatment groups on the student outcome)
• Dependent variable: students’ overall grade
• Independent variables:
- A factor variable indicating which of the four avatar types (black male, black female, white male, white female) or the real professor they were taught by
- Number of lessons passed
- Student characteristics (age, race, gender, level of education)

2. Do students engage more when they are taught by avatar instructors than when they are taught by the professor?

Method 1: OLS/logistic regressions
• Dependent variable: number of lessons completed (can be replaced with other measures of engagement)
• Independent variables:
- Whether they were taught by an avatar instructor or by the real professor
- Number of lessons passed
- Student discussion participation rate
- Whether the student completed their learning goal
- Whether the student upgraded to receive the certificate
- Whether the student achieved a badge on the course
- Attrition rate
- Student characteristics (age, race, gender, level of education)
- Interaction of whether they were taught by an avatar instructor and a student characteristic. This would test for the heterogeneous treatment effects across the different levels of student characteristic.

Method 2: OLS/logistic regression (This method allows us to see a more detailed effect of each of the engagement groups on the student outcome)
• Dependent variable: number of lessons completed (can be replaced with other measures of engagement)
• Independent variables:
- A factor variable indicating which of the four avatar types (black male, black female, white male, white female) or the real professor they were taught by
- Number of lessons passed
- Student discussion participation rate
- Whether the student completed their learning goal
- Whether the student upgraded to receive the certificate
- Whether the student achieved a badge on the course
- Attrition rate
- Student characteristics (age, race, gender, level of education)

3. Do students who are taught by an avatar instructor of the same race or gender as them perform better or have better engagement outcomes than those who are taught by an avatar instructor of a different race or gender?

Method 1: OLS/logistic regressions (limit the sample to those taught by avatars only)
• Dependent variable: a measure of students’ performance or engagement (refer to the previous regression provisions)
• Independent variables:
- Student’s race (gender)
- Instructor’s race (gender)
- Interaction of student race (gender) and instructor race (gender). This would test for the
- Other relevant measures of student performance/engagement
- Student characteristics (age, level of education)


Method 2: OLS/logistic regressions
• Dependent variable: a measure of students’ performance or engagement (refer to the previous regression provisions)
• Independent variables:
• - Student’s race
- Student’s gender
- Instructor’s race
- Instructor’s gender
- Interaction of student race and instructor race
- Interaction of student gender and instructor gender
- Other relevant measures of student performance/engagement
- Student characteristics (age, level of education)

2024-04-15

2024-05-06

Measure of performance:
• overall grade
• number of lessons passed

Measures of engagement:
• number of lessons completed
• whether the student completed their learning goal
• students’ discussion participation (by count or rate)
• students' feedback on the course
• students' rating on the course
• whether the student left a comment for course feedback
• the amount of time between the student's initial enrollment and when they completed the course (or stopped working on the course if not completed)
• whether the student upgraded to receive a paid certificate for the course
• whether the student achieved a badge in the course

This study will be conducted within the framework of Professor Ian Ayres’ course American Contract Law I offered by Coursera, and will attempt to assess the impact of various modes of teaching on learning and engagement.

Randomization done in office by a computer.

Individual student

No

N/A

At least 500 students

At least 100 students per group