The impact of psychometric matching on student outcomes and student satisfaction in online 1-1 learning environments

Last registered on January 09, 2020
View Trial History

Pre-Trial

Trial Information

General Information

Title
The impact of psychometric matching on student outcomes and student satisfaction in online 1-1 learning environments
RCT ID
AEARCTR-0004444
Initial registration date
July 14, 2019

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
July 15, 2019, 9:25 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
January 09, 2020, 3:56 AM EST

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Country
New Zealand
Region
Auckland

Primary Investigator

Name
Jamie Beaton
Affiliation
Crimson Education
Contact Primary Investigator

Other Primary Investigator(s)

PI Name
Jamie Beaton
PI Affiliation
Oxford University
Contact Investigator
PI Name
Lucy Bowes
PI Affiliation
Oxford University
Contact Investigator

Additional Trial Information

Status
On going
Start date
2019-06-24
End date
2020-12-10
Keywords
Education
Additional Keywords
JEL code(s)
Secondary IDs
Abstract
This randomized control trial evaluates the impact of psychometric matching on student outcomes and student satisfaction. All students and tutors in the study are evaluated based on a psychometric assessment known as HEXACO. From there, half of the students are matched to tutors based on an algorithm which evaluates the HEXACO traits of the student and the tutor and half of the students are matched randomly. Student self-reported session ratings of tutors is used as a measure of satisfaction and student SAT score improvement is used as a measure of change in outcomes. This trial takes place in an online school environment in the context of 1-1 synchronous learning via video call. Randomization occurs through a random number generator function which assigns people below a certain cut-off value (0.5) to one trial and those at or above the cut-off value to the other trial. Blinding occurs as students and tutors do not know if they are matched through psychometrics or not. Additional blinding occurs as the data analysis of results will be conducted blind as well.
External Link(s)
HEXACO assessment

Registration Citation

Citation
Beaton, Jamie, Jamie Beaton and Lucy Bowes. 2020. "The impact of psychometric matching on student outcomes and student satisfaction in online 1-1 learning environments ." AEA RCT Registry. January 09. https://doi.org/10.1257/rct.4444-1.1
Former Citation
Beaton, Jamie, Jamie Beaton and Lucy Bowes. 2020. "The impact of psychometric matching on student outcomes and student satisfaction in online 1-1 learning environments ." AEA RCT Registry. January 09. https://www.socialscienceregistry.org/trials/4444/history/60284
Sponsors & Partners
Experimental Details

Interventions

Intervention(s)
The intervention is assignment of online 1-1 student-tutor pairs based on HEXACO through a matching algorithm compared to a random match. HEXACO includes six factors, or dimensions: Honesty-Humility (H), Emotionality (E), Extraversion (X), Agreeableness (A), Conscientiousness (C), and Openness to Experience (O). Each factor has multiple sub-dimensions and is evaluated on a 0 - 5 scale after completion of a psychometric assessment which lasts approximately twenty minutes.
Intervention Start Date
2020-01-15
Intervention End Date
2020-07-15

Primary Outcomes

Primary Outcomes (end points)
SAT score improvement and student self-reported satisfaction score.
Primary Outcomes (explanation)
SAT score improvement will be constructed by measuring the initial SAT or ACT score of the student when they join the program through the online exam preparation tool. Any students with the ACT will be converted to the equivalent SAT score. The student's final SAT score within the six month period on the online exam preparation tool will be captured. The relative difference will be calculated (can be positive or negative). Student self-reported satisfaction score will be the mean of the ratings of all the student's reviews of a given tutor.

Secondary Outcomes

Secondary Outcomes (end points)
Cancellation Rate (Student)
Cancellation Rate (Tutor)
Secondary Outcomes (explanation)
The proportion of tutoring sessions the student cancels. This could be attributed to a lack of engagement on the part of the student or a perceived lack of usefulness on the part of the student. It could also capture disorganization.
The proportion of tutoring session the tutor cancels. This could be attributed to a lack of engagement on the part of the tutor. It could also capture disorganization.

Experimental Design

Experimental Design
Students will be enrolled into the trial over a rolling six month period. Students will participate in the trial over a six month period. Upon entry into the program, students will be assigned into either the control or intervention group based on a random number generator. Students in the control group will have no tutors matched to them based on the psychometric matching algorithm. Students in the intervention group will have all tutors matched to them based on the psychometric matching algorithm.
Experimental Design Details
Students will be enrolled into the trial over a rolling six month period. Students will participate in the trial over a six month period. Upon entry into the program, students will be assigned into either the control or intervention group based on a random number generator. Students in the control group will have no tutors matched to them based on the psychometric matching algorithm. Students in the intervention group will have all tutors matched to them based on the psychometric matching algorithm.

Tutors will consist of a combination of those already on the platform prior to the trial beginning and new tutors added into the program throughout the RCT period. Students, however, will consist entirely of new enrollments in the program. When a student or a tutor is enrolled in the program, they are prompted to download the application which has an in-app psychometric assessment. They need to complete this psychometric test in order to proceed with booking their sessions.

A single student may have multiple tutors throughout the RCT period in different subjects concurrently and may or may not take SAT or ACT tutoring. The SAT/ACT is used as a standardized baseline assessment for all students. Students will not be aware if they have been assigned to the control or the intervention group. Upon entry into the RCT, students will sit an SAT/ACT assessment (pre-test) and will then sit an additional SAT/ACT assessment after 6 months (post-test). Additionally, students will complete a rating of their tutor after every session throughout the six month period reviewing on a score of 0-5 with other additional features and ability to leave comments.

Randomization Method
Randomization done by excel random number generator function.
Randomization Unit
Individual randomization.
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
816 students
Sample size: planned number of observations
816 students
Sample size (or number of clusters) by treatment arms
408 students in control, 408 students in intervention group
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Group 1 mean: 4.7 +- 0.5 Group 2 mean: 2.5% Power: 0.8 Alpha: 0.05 Beta: 0.2 Suggested Sample Size: 546
Supporting Documents and Materials
IRB

Institutional Review Boards (IRBs)

IRB Name
Blavatnik School of Government's Departmental Research Ethics Committee
IRB Approval Date
2019-12-03
IRB Approval Number
SSD/CUREC1A/BSG_C1A-19-04
Analysis Plan

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials