|
Field
Primary Outcomes (End Points)
|
Before
Measure of performance:
• overall grade
• number of lessons passed
Measures of engagement:
• number of lessons completed
• whether the student completed their learning goal
• students’ discussion participation (by count or rate)
• students' feedback on the course
• students' rating on the course
• whether the student left a comment for course feedback
• the amount of time between the student's initial enrollment and when they completed the course (or stopped working on the course if not completed)
• whether the student upgraded to receive a paid certificate for the course
• whether the student achieved a badge in the course
|
After
Measure of performance:
• overall grade
• number of lessons passed
Measures of engagement:
• number of lessons completed
• student goal
• whether the student completed their learning goal
• whether the student upgraded to receive a paid certificate for the course
whether the student achieved a badge in the course
|
|
Field
Intervention (Hidden)
|
Before
Students of a Coursera Contracts course will be randomly assigned to one of the five groups. In each group, the substantive content will remain the same – same words spoken and same words shown in powerpoint slides and photos, but the groups will differ with the visual image and voice of the professor. Here is a description of the specific groups:
• Control group: Students in this group will learn the course material by being shown the preexisting videos with the professor delivering the material throughout the course
• Treatment group 1: Students in this group will learn the course by being shown videos in which the professor is replaced by a black male avatar instructor.
• Treatment group 2: Students in this group will learn the course by being shown videos in which the professor is replaced by a black female avatar instructor.
• Treatment group 3: Students in this group will learn the course by being shown videos in which the professor is replaced by a white male avatar instructor.
• Treatment group 4: Students in this group will learn the course by being shown videos in which the professor is replaced by a white female avatar instructor.
The following list presents specific questions of interest along with the appropriate statistical approaches to studying them:
1. Do students perform better when they are taught by avatar instructors than when they are taught by the professor?
Method 1: OLS regression
• Dependent variable: students’ overall grade
• Independent variables:
- Whether they were taught by an avatar instructor or by the real professor
- Number of lessons passed
- Student characteristics (age, race, gender, level of education)
Method 2: OLS regression (This method allows us to see a more detailed effect of each of the treatment groups on the student outcome)
• Dependent variable: students’ overall grade
• Independent variables:
- A factor variable indicating which of the four avatar types (black male, black female, white male, white female) or the real professor they were taught by
- Number of lessons passed
- Student characteristics (age, race, gender, level of education)
2. Do students engage more when they are taught by avatar instructors than when they are taught by the professor?
Method 1: OLS/logistic regressions
• Dependent variable: number of lessons completed (can be replaced with other measures of engagement)
• Independent variables:
- Whether they were taught by an avatar instructor or by the real professor
- Number of lessons passed
- Student discussion participation rate
- Whether the student completed their learning goal
- Whether the student upgraded to receive the certificate
- Whether the student achieved a badge on the course
- Attrition rate
- Student characteristics (age, race, gender, level of education)
- Interaction of whether they were taught by an avatar instructor and a student characteristic. This would test for the heterogeneous treatment effects across the different levels of student characteristic.
Method 2: OLS/logistic regression (This method allows us to see a more detailed effect of each of the engagement groups on the student outcome)
• Dependent variable: number of lessons completed (can be replaced with other measures of engagement)
• Independent variables:
- A factor variable indicating which of the four avatar types (black male, black female, white male, white female) or the real professor they were taught by
- Number of lessons passed
- Student discussion participation rate
- Whether the student completed their learning goal
- Whether the student upgraded to receive the certificate
- Whether the student achieved a badge on the course
- Attrition rate
- Student characteristics (age, race, gender, level of education)
3. Do students who are taught by an avatar instructor of the same race or gender as them perform better or have better engagement outcomes than those who are taught by an avatar instructor of a different race or gender?
Method 1: OLS/logistic regressions (limit the sample to those taught by avatars only)
• Dependent variable: a measure of students’ performance or engagement (refer to the previous regression provisions)
• Independent variables:
- Student’s race (gender)
- Instructor’s race (gender)
- Interaction of student race (gender) and instructor race (gender). This would test for the
- Other relevant measures of student performance/engagement
- Student characteristics (age, level of education)
Method 2: OLS/logistic regressions
• Dependent variable: a measure of students’ performance or engagement (refer to the previous regression provisions)
• Independent variables:
• - Student’s race
- Student’s gender
- Instructor’s race
- Instructor’s gender
- Interaction of student race and instructor race
- Interaction of student gender and instructor gender
- Other relevant measures of student performance/engagement
- Student characteristics (age, level of education)
|
After
Students of a Coursera Contracts course will be randomly assigned to one of the five groups. In each group, the substantive content will remain the same – same words spoken and same words shown in powerpoint slides and photos, but the groups will differ with the visual image and voice of the professor. Here is a description of the specific groups:
• Control group: Students in this group will learn the course material by being shown the preexisting videos with the professor delivering the material throughout the course
• Treatment group 1: Students in this group will learn the course by being shown videos in which the professor is replaced by a black male avatar instructor.
• Treatment group 2: Students in this group will learn the course by being shown videos in which the professor is replaced by a black female avatar instructor.
• Treatment group 3: Students in this group will learn the course by being shown videos in which the professor is replaced by a white male avatar instructor.
• Treatment group 4: Students in this group will learn the course by being shown videos in which the professor is replaced by a white female avatar instructor.
The following list presents specific questions of interest along with the appropriate statistical approaches to studying them:
1. Do students perform better or have better engagement when they are taught by avatar instructors than when they are taught by the professor?
Method 1: OLS regression examining the effect of the type of instructor (whether they were taught by any one of the avatars or by the real professor) on students’ overall grade (or the number of items passed)
(1) Without student controls
• Dependent variable: students’ overall grade (or any of the other outcome variables)
• Independent variable: indicator if they were taught by an avatar instructor
(2) With student demographic (age, race, gender, level of education) controls
Method 2: OLS regression examining the effect of the specific type of instructor (black male avatar, black female avatar, white male avatar, white female avatar, or real professor) on students’ overall grade (or the number of items passed). This method allows us to see a more detailed effect of each of the treatment groups on the student outcome.
(1) Without student controls
• Dependent variable: students’ overall grade (or any of the other outcome variables)
• Independent variables:
- indicators for each of the four avatar types (black male, black female, white male, white female)
(2) With student demographic (age, race, gender, level of education) controls
(3) To observe heterogeneous treatment effects:
• Dependent variable: students’ overall grade (or any of the outcome variables)
• Independent variables:
- indicators for each of the four avatar types (black male, black female, white male, white female)
- Uninteracted student demographic indicators (age, race, gender, level of education categories)
- Interaction of the avatar indicators with the student demographic category indicators
*** We will use interact coefficients to test whether particular types of students had superior outcomes with particular avatars
(4) Perform post-estimation (f-test) to see if each treatment effect various across different subgroups of student characteristics.
*** Significant coefficients here would indicate that the effect of instructor type differs across different student characteristic groups when it comes to students’ overall grades (or the number of items passed) as a measure of performance
3. Do students who are taught by an avatar instructor of the same race or gender as them perform better or have better engagement outcomes than those who are taught by an avatar instructor of a different race or gender?
Method 1: Regression examining the effect of students’ race (or gender) and the instructor’s race (or gender) on students’ overall grades (or the number of items passed), a measure of performance
(1) Without student controls:
• Dependent variable: students’ overall grade (or any of the outcome variables)
• Independent variables:
- Student’s race (gender)
- Instructor’s race (gender)
- Interaction of the student race and instructor race
(2) With student controls. This model also accounts for the heterogeneous treatment effects.
• Dependent variable: students’ overall grade (or any of the outcome variables)
• Independent variables:
- Student’s race (gender)
- Instructor’s race (gender)
- Interaction of the student race (gender) and instructor race (gender)
- Student characteristics (age, gender/race, level of education)
(3) Perform post-estimation (f-test) to see if each treatment effect various across different subgroups of student characteristics.
*** Significant coefficients here would indicate that the effect of instructor type differs across different student characteristic groups when it comes to each measure of performance
Method 2: Regression examining the effect of students’ race (or gender) and the instructor’s race (or gender) on students’ number of lessons completed (or whether the student completed their learning goal), a measure of engagement
(1) Without student controls:
• Dependent variable: the number of lessons completed (or whether the student completed their learning goal)
• Independent variables:
- Student’s race (gender)
- Instructor’s race (gender)
- Interaction of the student race and instructor race
(2) With student controls. This model also accounts for the heterogeneous treatment effects.
• Dependent variable: the number of lessons completed (or whether the student completed their learning goal)
• Independent variables:
- Student’s race (gender)
- Instructor’s race (gender)
- Interaction of the student race (gender) and instructor race (gender)
- Student characteristics (age, gender/race, level of education)
(3) Perform post-estimation (f-test) to see if each treatment effect various across different subgroups of student characteristics.
*** Significant coefficients here would indicate that the effect of instructor type differs across different student characteristic groups when it comes to each measure of engagement
|