Experimental Design Details
The following list presents specific questions of interest along with the appropriate statistical approaches to studying them:
1. Do students perform better or have better engagement when they are taught by avatar instructors than when they are taught by the professor?
Method 1: OLS regression examining the effect of the type of instructor (whether they were taught by any one of the avatars or by the real professor) on students’ overall grade (or other relevant outcome variables)
(1) Without student controls
• Dependent variable: students’ overall grade (or other relevant outcome variables)
• Independent variable: indicator if they were taught by an avatar instructor
(2) With student demographic (age, race, gender, level of education) controls
Method 2: OLS regression examining the effect of the specific type of instructor (black male avatar, black female avatar, white male avatar, white female avatar, or real professor) on students’ overall grade (or other relevant outcome variables). This method allows us to see a more detailed effect of each of the treatment groups on the student outcome.
(1) Without student controls
• Dependent variable: students’ overall grade (or other relevant outcome variables)
• Independent variables:
- indicators for each of the four avatar types (black male, black female, white male, white female)
(2) With student demographic (age, race, gender, level of education) controls
(3) To observe heterogeneous treatment effects:
• Dependent variable: students’ overall grade (or other relevant outcome variables)
• Independent variables:
- indicators for each of the four avatar types (black male, black female, white male, white female)
- Uninteracted student demographic indicators (age, race, gender, level of education categories)
- Interaction of the avatar indicators with the student demographic category indicators
*** We will use interact coefficients to test whether particular types of students had superior outcomes with particular avatars
(4) Perform post-estimation (f-test) to see if each treatment effect various across different subgroups of student characteristics.
*** Significant coefficients here would indicate that the effect of instructor type differs across different student characteristic groups when it comes to students’ overall grades (or other relevant outcome variables) as a measure of performance
3. Do students who are taught by an avatar instructor of the same race or gender as them perform better or have better engagement outcomes than those who are taught by an avatar instructor of a different race or gender?
Method 1: Regression examining the effect of students’ race (or gender) and the instructor’s race (or gender) on students’ overall grades (or other relevant outcome variables), a measure of performance
(1) Without student controls:
• Dependent variable: students’ overall grade (or other relevant outcome variables)
• Independent variables:
- Student’s race (gender)
- Instructor’s race (gender)
- Interaction of the student race and instructor race
(2) With student controls. This model also accounts for the heterogeneous treatment effects.
• Dependent variable: students’ overall grade (or other relevant outcome variables)
• Independent variables:
- Student’s race (gender)
- Instructor’s race (gender)
- Interaction of the student race (gender) and instructor race (gender)
- Student characteristics (age, gender/race, level of education)
(3) Perform post-estimation (f-test) to see if each treatment effect various across different subgroups of student characteristics.
*** Significant coefficients here would indicate that the effect of instructor type differs across different student characteristic groups when it comes to each measure of performance
Method 2: Regression examining the effect of students’ race (or gender) and the instructor’s race (or gender) on students’ number of lessons completed (or other relevant outcome variables), a measure of engagement
(1) Without student controls:
• Dependent variable: the number of lessons completed (or other relevant outcome variables)
• Independent variables:
- Student’s race (gender)
- Instructor’s race (gender)
- Interaction of the student race and instructor race
(2) With student controls. This model also accounts for the heterogeneous treatment effects.
• Dependent variable: the number of lessons completed (or other relevant outcome variables)
• Independent variables:
- Student’s race (gender)
- Instructor’s race (gender)
- Interaction of the student race (gender) and instructor race (gender)
- Student characteristics (age, gender/race, level of education)
(3) Perform post-estimation (f-test) to see if each treatment effect various across different subgroups of student characteristics.
*** Significant coefficients here would indicate that the effect of instructor type differs across different student characteristic groups when it comes to each measure of engagement