Back to History Current Version

The impact of psychometric matching on student outcomes and student satisfaction in online 1-1 learning environments

Last registered on July 15, 2019
View Trial History

Pre-Trial

Trial Information

General Information

Title
The impact of psychometric matching on student outcomes and student satisfaction in online 1-1 learning environments
RCT ID
AEARCTR-0004444
Initial registration date
July 14, 2019

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
July 15, 2019, 9:25 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Country
New Zealand
Region
Auckland

Primary Investigator

Name
Jamie Beaton
Affiliation
Crimson Education
Contact Primary Investigator

Other Primary Investigator(s)

Additional Trial Information

Status
On going
Start date
2019-06-24
End date
2020-03-24
Keywords
Education
Additional Keywords
JEL code(s)
Secondary IDs
Abstract
This randomized control trial evaluates the impact of psychometric matching on student outcomes and student satisfaction. All students and tutors in the study are evaluated based on a psychometric assessment known as HEXACO. From there, half of the students are matched to tutors based on an algorithm which evaluates the HEXACO traits of the student and the tutor and half of the students are matched randomly. Student self-reported session ratings of tutors is used as a measure of satisfaction and student SAT score improvement is used as a measure of change in outcomes.
External Link(s)

Registration Citation

Citation
Beaton, Jamie. 2019. "The impact of psychometric matching on student outcomes and student satisfaction in online 1-1 learning environments ." AEA RCT Registry. July 15. https://doi.org/10.1257/rct.4444-1.0
Former Citation
Beaton, Jamie. 2019. "The impact of psychometric matching on student outcomes and student satisfaction in online 1-1 learning environments ." AEA RCT Registry. July 15. https://www.socialscienceregistry.org/trials/4444/history/50022
Sponsors & Partners
Experimental Details

Interventions

Intervention(s)
The intervention is assignment of online 1-1 student-tutor pairs based on HEXACO through a matching algorithm compared to a random match. HEXACO includes six factors, or dimensions: Honesty-Humility (H), Emotionality (E), Extraversion (X), Agreeableness (A), Conscientiousness (C), and Openness to Experience (O). Each factor has multiple sub-dimensions and is evaluated on a 0 - 5 scale after completion of a psychometric assessment which lasts approximately twenty minutes.
Intervention Start Date
2019-06-24
Intervention End Date
2019-09-24

Primary Outcomes

Primary Outcomes (end points)
SAT score improvement and student self-reported satisfaction score.
Primary Outcomes (explanation)
SAT score improvement will be constructed by measuring the initial SAT or ACT score of the student when they join the program through the online exam preparation tool. Any students with the ACT will be converted to the equivalent SAT score. The student's final SAT score within the six month period on the online exam preparation tool will be captured. The relative difference will be calculated (can be positive or negative). Student self-reported satisfaction score will be the mean of the ratings of all the student's reviews of a given tutor.

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
Students will be enrolled into the trial over a rolling three month period. Students will participate in the trial over a six month period. Upon entry into the program, students will be assigned into either the control or intervention group based on a random number generator. Students in the control group will have no tutors matched to them based on the psychometric matching algorithm. Students in the intervention group will have all tutors matched to them based on the psychometric matching algorithm.
Experimental Design Details
Students will be enrolled into the trial over a rolling three month period. Students will participate in the trial over a six month period. Upon entry into the program, students will be assigned into either the control or intervention group based on a random number generator. Students in the control group will have no tutors matched to them based on the psychometric matching algorithm. Students in the intervention group will have all tutors matched to them based on the psychometric matching algorithm.

Tutors will consist of a combination of those already on the platform prior to the trial beginning and new tutors added into the program throughout the RCT period. Students, however, will consist entirely of new enrollments in the program. When a student or a tutor is enrolled in the program, they are prompted to download the application which has an in-app psychometric assessment.

A single student may have multiple tutors throughout the RCT period in different subjects concurrently and may or may not take SAT or ACT tutoring. The SAT/ACT is used as a standardized baseline assessment for all students. Students will not be aware if they have been assigned to the control or the intervention group.
Randomization Method
Randomization done by excel random number generator function.
Randomization Unit
Individual randomization.
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
300 students
Sample size: planned number of observations
300 students
Sample size (or number of clusters) by treatment arms
150 students in control, 150 students in intervention group
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Supporting Documents and Materials
IRB

Institutional Review Boards (IRBs)

IRB Name
IRB Approval Date
IRB Approval Number
Analysis Plan

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials