The impact of psychometric matching on student outcomes and student satisfaction in online 1-1 learning environments

The intervention is assignment of online 1-1 student-tutor pairs based on HEXACO through a matching algorithm compared to a random match. HEXACO includes six factors, or dimensions: Honesty-Humility (H), Emotionality (E), Extraversion (X), Agreeableness (A), Conscientiousness (C), and Openness to Experience (O). Each factor has multiple sub-dimensions and is evaluated on a 0 - 5 scale after completion of a psychometric assessment which lasts approximately twenty minutes.

Our first analysis will evaluate all student and tutor rating data and all available HEXACO data following a randomized control trial in which half of the student population are allocated to HEXACO matching and other are not over a six month period. We will run a regression analysis with self-reported student rating of teacher as the independent variable and the psychometric characteristics within HEXACO in various forms as the dependent variables. The first regression will utilize student psychometric characteristics as the dependent variable. The second regression will utilize tutor psychometric characteristics as the dependent variable. The third regression will utilize the absolute difference (magnitude) between the student and tutor psychometric characteristics as the dependent variable. The fourth regression will utilize the difference between the student and tutor psychometric characteristics as the dependent variable. We will additionally perform a random forest analysis to attempt to uncover any other relationships that may be of interest which predict session rating with attention paid to the risk of spurious relationships from data-mining. We note that only the regressions that look at the relationship between student and tutor psychometric characteristics and student session ratings of tutors are actually measuring a potential matching framework. The prior regressions are measuring only whether psychometric scores of the student can predict any statistical difference in their propensity to rank and whether psychometric scores of the tutor can predict any statistical difference in the scores they are likely to receive.

To all of these controls, we add dummy variables to account for age of the student, country of the student, country of the tutor and pay-rate of the tutor. We add these controls in order to disentangle effects that could confound our results. A possible mechanism would be if matching students and tutors by country of origin is actually the most effective and certain countries of origin are highly correlated with certain psychometric profiles. In this example, we may see a strong relationship between matching students and tutors based on similar psychometric profiles but this may be explained by the lurking variable of countries of origin.

Our second analysis will evaluate all student score improvement data and all available HEXACO data following a randomized control trial in which half of the student population are allocated to HEXACO matching and other are not over a six month period. We will run a regression analysis with student SAT score improvement as the independent variable and the psychometric characteristics within HEXACO in various forms as the dependent variables. The first regression will utilize student psychometric characteristics as the dependent variable. The second regression will utilize tutor psychometric characteristics as the dependent variable. The third regression will utilize the absolute difference (magnitude) between the student and tutor psychometric characteristics as the dependent variable. The fourth regression will utilize the difference between the student and tutor psychometric characteristics as the dependent variable. We will additionally perform a random forest analysis to attempt to uncover any other relationships that may be of interest which predict session rating with attention paid to the risk of spurious relationships from data-mining. We note that only the regressions that look at the relationship between student and tutor psychometric characteristics and SAT score improvement are actually measuring a potential matching framework. The prior regressions are measuring only whether psychometric scores of the student can predict any statistical difference in their SAT score improvement and whether psychometric scores of the tutor can predict any statistical difference in the statistical difference in the SAT score improvement of the student. While it may be the case that certain psychometric characteristics predict a tutor is more effective, this does not provide insight into how to most effectively optimize student-tutor matching given a pool of potential tutors. We note that it is likely based on our literature review that psychometric characteristics can indeed predict which tutors are likely to be more effective or which students are more likely to be more effective learners.

We additionally wish to compare the statistical relationship between SAT student score improvement and average session rating. This enables us to extract a relationship between our variables for student satisfaction and session outcomes. In order to do this, we run a regression of SAT student score improvement against average student reported session rating of their SAT or ACT tutor. As a reminder, we normalize all ACT scores into equivalent SAT scores for the purposes of our analysis. We also run a regression of SAT student score improvement against average student reported session rating across all of their tutors. Finally, we run a regression of SAT student score improvement across average student reported session rating of their SAT or ACT tutor, normalized by the average session rating of their other tutors to correct for rating bias by the student.

We will capture the effect of country through multi-level modeling with a two-stage hierarchical regression to capture country effects on the students.

2020-01-15

2020-07-15

SAT score improvement and student self-reported satisfaction score.

SAT score improvement will be constructed by measuring the initial SAT or ACT score of the student when they join the program through the online exam preparation tool. Any students with the ACT will be converted to the equivalent SAT score. The student's final SAT score within the six month period on the online exam preparation tool will be captured. The relative difference will be calculated (can be positive or negative). Student self-reported satisfaction score will be the mean of the ratings of all the student's reviews of a given tutor.

Cancellation Rate (Student)
Cancellation Rate (Tutor)

The proportion of tutoring sessions the student cancels. This could be attributed to a lack of engagement on the part of the student or a perceived lack of usefulness on the part of the student. It could also capture disorganization.
The proportion of tutoring session the tutor cancels. This could be attributed to a lack of engagement on the part of the tutor. It could also capture disorganization.

Students will be enrolled into the trial over a rolling six month period. Students will participate in the trial over a six month period. Upon entry into the program, students will be assigned into either the control or intervention group based on a random number generator. Students in the control group will have no tutors matched to them based on the psychometric matching algorithm. Students in the intervention group will have all tutors matched to them based on the psychometric matching algorithm.

Students will be enrolled into the trial over a rolling six month period. Students will participate in the trial over a six month period. Upon entry into the program, students will be assigned into either the control or intervention group based on a random number generator. Students in the control group will have no tutors matched to them based on the psychometric matching algorithm. Students in the intervention group will have all tutors matched to them based on the psychometric matching algorithm.

Tutors will consist of a combination of those already on the platform prior to the trial beginning and new tutors added into the program throughout the RCT period. Students, however, will consist entirely of new enrollments in the program. When a student or a tutor is enrolled in the program, they are prompted to download the application which has an in-app psychometric assessment. They need to complete this psychometric test in order to proceed with booking their sessions.

A single student may have multiple tutors throughout the RCT period in different subjects concurrently and may or may not take SAT or ACT tutoring. The SAT/ACT is used as a standardized baseline assessment for all students. Students will not be aware if they have been assigned to the control or the intervention group. Upon entry into the RCT, students will sit an SAT/ACT assessment (pre-test) and will then sit an additional SAT/ACT assessment after 6 months (post-test). Additionally, students will complete a rating of their tutor after every session throughout the six month period reviewing on a score of 0-5 with other additional features and ability to leave comments.

Randomization done by excel random number generator function.

Individual randomization.

Yes

816 students

816 students

408 students in control, 408 students in intervention group

Group 1 mean: 4.7 +- 0.5 Group 2 mean: 2.5% Power: 0.8 Alpha: 0.05 Beta: 0.2 Suggested Sample Size: 546