Using Predictive Analytics to Track Students: Evidence from a Seven-College Experiment
Last registered on June 15, 2021

Pre-Trial

Trial Information
General Information
Title
Using Predictive Analytics to Track Students: Evidence from a Seven-College Experiment
RCT ID
AEARCTR-0007827
Initial registration date
June 15, 2021
Last updated
June 15, 2021 2:30 PM EDT
Location(s)
Primary Investigator
Affiliation
Columbia University Teachers College
Other Primary Investigator(s)
PI Affiliation
Teachers College, Columbia University
PI Affiliation
Teachers College, Columbia University
Additional Trial Information
Status
Completed
Start date
2016-05-01
End date
2021-06-01
Secondary IDs
Abstract
Tracking is widespread in U.S. education. In post-secondary education alone, at least 71% of colleges use a test to track students. However, there are concerns that the most frequently used college placement exams lack validity and reliability, and unnecessarily place students from under-represented groups into remedial courses. While recent research has shown that tracking can have positive effects on student learning, inaccurate placement has consequences: students face misaligned curricula and must pay tuition for remedial courses that do not bear credits toward graduation. We develop an alternative system to place students that uses predictive analytics to combine multiple measures into a placement instrument. Compared to colleges’ existing placement tests, the algorithm is more predictive of future performance. We then conduct an experiment across seven colleges to evaluate the algorithm’s effects on students. Placement rates into college-level courses increased substantially without reducing pass rates. Adjusting for multiple testing, algorithmic placement generally, though not always, narrowed gaps in college placement rates and remedial course taking across demographic groups. A detailed cost analysis shows that the algorithmic placement system is socially efficient: it saves costs for students while increasing college credits earned, which more than offsets increased costs for colleges. Costs could be reduced with improved data digitization, as opposed to entering data by hand.
External Link(s)
Registration Citation
Citation
BERGMAN, PETER, Elizabeth Kopko and Julio Rodriguez. 2021. "Using Predictive Analytics to Track Students: Evidence from a Seven-College Experiment." AEA RCT Registry. June 15. https://doi.org/10.1257/rct.7827-1.0.
Experimental Details
Interventions
Intervention(s)
See paper: http://www.columbia.edu/~psb2101/bergmananalytics.pdf
Intervention Start Date
2016-08-01
Intervention End Date
2018-12-31
Primary Outcomes
Primary Outcomes (end points)
See paper.
Primary Outcomes (explanation)
See paper.
Secondary Outcomes
Secondary Outcomes (end points)
Secondary Outcomes (explanation)
Experimental Design
Experimental Design
See paper: http://www.columbia.edu/~psb2101/bergmananalytics.pdf
Experimental Design Details
Randomization Method
Computer
Randomization Unit
Individual
Was the treatment clustered?
No
Experiment Characteristics
Sample size: planned number of clusters
n/a
Sample size: planned number of observations
12,544
Sample size (or number of clusters) by treatment arms
6,141 control 6,403 treatment
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
2 percent point increase in course pass rates.
IRB
INSTITUTIONAL REVIEW BOARDS (IRBs)
IRB Name
Teachers College, Columbia University
IRB Approval Date
2016-04-08
IRB Approval Number
14-361
Post-Trial
Post Trial Information
Study Withdrawal
Intervention
Is the intervention completed?
Yes
Intervention Completion Date
December 01, 2018, 12:00 AM +00:00
Is data collection complete?
No
Data Publication
Data Publication
Is public data available?
No
Program Files
Program Files
Reports, Papers & Other Materials
Relevant Paper(s)
REPORTS & OTHER MATERIALS