|
Field
Abstract
|
Before
This randomized control trial evaluates the impact of psychometric matching on student outcomes and student satisfaction. All students and tutors in the study are evaluated based on a psychometric assessment known as HEXACO. From there, half of the students are matched to tutors based on an algorithm which evaluates the HEXACO traits of the student and the tutor and half of the students are matched randomly. Student self-reported session ratings of tutors is used as a measure of satisfaction and student SAT score improvement is used as a measure of change in outcomes.
|
After
This randomized control trial evaluates the impact of psychometric matching on student outcomes and student satisfaction. All students and tutors in the study are evaluated based on a psychometric assessment known as HEXACO. From there, half of the students are matched to tutors based on an algorithm which evaluates the HEXACO traits of the student and the tutor and half of the students are matched randomly. Student self-reported session ratings of tutors is used as a measure of satisfaction and student SAT score improvement is used as a measure of change in outcomes. This trial takes place in an online school environment in the context of 1-1 synchronous learning via video call. Randomization occurs through a random number generator function which assigns people below a certain cut-off value (0.5) to one trial and those at or above the cut-off value to the other trial. Blinding occurs as students and tutors do not know if they are matched through psychometrics or not. Additional blinding occurs as the data analysis of results will be conducted blind as well.
|
|
Field
Trial End Date
|
Before
March 24, 2020
|
After
December 10, 2020
|
|
Field
Last Published
|
Before
July 15, 2019 09:25 AM
|
After
January 09, 2020 03:56 AM
|
|
Field
Intervention Start Date
|
Before
June 24, 2019
|
After
January 15, 2020
|
|
Field
Intervention End Date
|
Before
September 24, 2019
|
After
July 15, 2020
|
|
Field
Experimental Design (Public)
|
Before
Students will be enrolled into the trial over a rolling three month period. Students will participate in the trial over a six month period. Upon entry into the program, students will be assigned into either the control or intervention group based on a random number generator. Students in the control group will have no tutors matched to them based on the psychometric matching algorithm. Students in the intervention group will have all tutors matched to them based on the psychometric matching algorithm.
|
After
Students will be enrolled into the trial over a rolling six month period. Students will participate in the trial over a six month period. Upon entry into the program, students will be assigned into either the control or intervention group based on a random number generator. Students in the control group will have no tutors matched to them based on the psychometric matching algorithm. Students in the intervention group will have all tutors matched to them based on the psychometric matching algorithm.
|
|
Field
Was the treatment clustered?
|
Before
No
|
After
Yes
|
|
Field
Planned Number of Clusters
|
Before
300 students
|
After
816 students
|
|
Field
Planned Number of Observations
|
Before
300 students
|
After
816 students
|
|
Field
Sample size (or number of clusters) by treatment arms
|
Before
150 students in control, 150 students in intervention group
|
After
408 students in control, 408 students in intervention group
|
|
Field
Power calculation: Minimum Detectable Effect Size for Main Outcomes
|
Before
|
After
Group 1 mean: 4.7 +- 0.5
Group 2 mean: 2.5%
Power: 0.8
Alpha: 0.05
Beta: 0.2
Suggested Sample Size: 546
|
|
Field
Intervention (Hidden)
|
Before
Our next analysis will evaluate all student score improvement data and all available HEXACO data following a randomized control trial in which half of the student population are allocated to HEXACO matching and other are not over a six month period. We will run a regression analysis with student SAT score improvement as the independent variable and the psychometric characteristics within HEXACO in various forms as the dependent variables. The first regression will utilize student psychometric characteristics as the dependent variable. The second regression will utilize tutor psychometric characteristics as the dependent variable. The third regression will utilize the absolute difference (magnitude) between the student and tutor psychometric characteristics as the dependent variable. The fourth regression will utilize the difference between the student and tutor psychometric characteristics as the dependent variable. We will additionally perform a random forest analysis to attempt to uncover any other relationships that may be of interest which predict session rating with attention paid to the risk of spurious relationships from data-mining. We note that only the regressions that look at the relationship between student and tutor psychometric characteristics and SAT score improvement are actually measuring a potential matching framework. The prior regressions are measuring only whether psychometric scores of the student can predict any statistical difference in their SAT score improvement and whether psychometric scores of the tutor can predict any statistical difference in the statistical difference in the SAT score improvement of the student. While it may be the case that certain psychometric characteristics predict a tutor is more effective, this does not provide insight into how to most effectively optimize student-tutor matching given a pool of potential tutors. We note that it is likely based on our literature review that psychometric characteristics can indeed predict which tutors are likely to be more effective or which students are more likely to be more effective learners.
We additionally wish to compare the statistical relationship between SAT student score improvement and average session rating. This enables us to extract a relationship between our variables for student satisfaction and session outcomes. In order to do this, we run a regression of SAT student score improvement against average student reported session rating of their SAT or ACT tutor. As a reminder, we normalize all ACT scores into equivalent SAT scores for the purposes of our analysis. We also run a regression of SAT student score improvement against average student reported session rating across all of their tutors. Finally, we run a regression of SAT student score improvement across average student reported session rating of their SAT or ACT tutor, normalized by the average session rating of their other tutors to correct for rating bias by the student.
|
After
Our first analysis will evaluate all student and tutor rating data and all available HEXACO data following a randomized control trial in which half of the student population are allocated to HEXACO matching and other are not over a six month period. We will run a regression analysis with self-reported student rating of teacher as the independent variable and the psychometric characteristics within HEXACO in various forms as the dependent variables. The first regression will utilize student psychometric characteristics as the dependent variable. The second regression will utilize tutor psychometric characteristics as the dependent variable. The third regression will utilize the absolute difference (magnitude) between the student and tutor psychometric characteristics as the dependent variable. The fourth regression will utilize the difference between the student and tutor psychometric characteristics as the dependent variable. We will additionally perform a random forest analysis to attempt to uncover any other relationships that may be of interest which predict session rating with attention paid to the risk of spurious relationships from data-mining. We note that only the regressions that look at the relationship between student and tutor psychometric characteristics and student session ratings of tutors are actually measuring a potential matching framework. The prior regressions are measuring only whether psychometric scores of the student can predict any statistical difference in their propensity to rank and whether psychometric scores of the tutor can predict any statistical difference in the scores they are likely to receive.
To all of these controls, we add dummy variables to account for age of the student, country of the student, country of the tutor and pay-rate of the tutor. We add these controls in order to disentangle effects that could confound our results. A possible mechanism would be if matching students and tutors by country of origin is actually the most effective and certain countries of origin are highly correlated with certain psychometric profiles. In this example, we may see a strong relationship between matching students and tutors based on similar psychometric profiles but this may be explained by the lurking variable of countries of origin.
Our second analysis will evaluate all student score improvement data and all available HEXACO data following a randomized control trial in which half of the student population are allocated to HEXACO matching and other are not over a six month period. We will run a regression analysis with student SAT score improvement as the independent variable and the psychometric characteristics within HEXACO in various forms as the dependent variables. The first regression will utilize student psychometric characteristics as the dependent variable. The second regression will utilize tutor psychometric characteristics as the dependent variable. The third regression will utilize the absolute difference (magnitude) between the student and tutor psychometric characteristics as the dependent variable. The fourth regression will utilize the difference between the student and tutor psychometric characteristics as the dependent variable. We will additionally perform a random forest analysis to attempt to uncover any other relationships that may be of interest which predict session rating with attention paid to the risk of spurious relationships from data-mining. We note that only the regressions that look at the relationship between student and tutor psychometric characteristics and SAT score improvement are actually measuring a potential matching framework. The prior regressions are measuring only whether psychometric scores of the student can predict any statistical difference in their SAT score improvement and whether psychometric scores of the tutor can predict any statistical difference in the statistical difference in the SAT score improvement of the student. While it may be the case that certain psychometric characteristics predict a tutor is more effective, this does not provide insight into how to most effectively optimize student-tutor matching given a pool of potential tutors. We note that it is likely based on our literature review that psychometric characteristics can indeed predict which tutors are likely to be more effective or which students are more likely to be more effective learners.
We additionally wish to compare the statistical relationship between SAT student score improvement and average session rating. This enables us to extract a relationship between our variables for student satisfaction and session outcomes. In order to do this, we run a regression of SAT student score improvement against average student reported session rating of their SAT or ACT tutor. As a reminder, we normalize all ACT scores into equivalent SAT scores for the purposes of our analysis. We also run a regression of SAT student score improvement against average student reported session rating across all of their tutors. Finally, we run a regression of SAT student score improvement across average student reported session rating of their SAT or ACT tutor, normalized by the average session rating of their other tutors to correct for rating bias by the student.
We will capture the effect of country through multi-level modeling with a two-stage hierarchical regression to capture country effects on the students.
|
|
Field
Did you obtain IRB approval for this study?
|
Before
No
|
After
Yes
|
|
Field
Secondary Outcomes (End Points)
|
Before
|
After
Cancellation Rate (Student)
Cancellation Rate (Tutor)
|
|
Field
Secondary Outcomes (Explanation)
|
Before
|
After
The proportion of tutoring sessions the student cancels. This could be attributed to a lack of engagement on the part of the student or a perceived lack of usefulness on the part of the student. It could also capture disorganization.
The proportion of tutoring session the tutor cancels. This could be attributed to a lack of engagement on the part of the tutor. It could also capture disorganization.
|
|
Field
Pi as first author
|
Before
No
|
After
Yes
|