Using Predictive Analytics to Track Students: Evidence from a Seven-College Experiment

Last registered on June 15, 2021

Pre-Trial

Trial Information

General Information

Title
Using Predictive Analytics to Track Students: Evidence from a Seven-College Experiment
RCT ID
AEARCTR-0007827
Initial registration date
June 15, 2021
Last updated
June 15, 2021, 2:30 PM EDT

Locations

Primary Investigator

Affiliation
Columbia University Teachers College

Other Primary Investigator(s)

PI Affiliation
Teachers College, Columbia University
PI Affiliation
Teachers College, Columbia University

Additional Trial Information

Status
Completed
Start date
2016-05-01
End date
2021-06-01
Secondary IDs
Abstract
Tracking is widespread in U.S. education. In post-secondary education alone, at least 71% of colleges use a test to track students. However, there are concerns that the most frequently used college placement exams lack validity and reliability, and unnecessarily place students from under-represented groups into remedial courses. While recent research has shown that tracking can have positive effects on student learning, inaccurate placement has consequences: students face misaligned curricula and must pay tuition for remedial courses that do not bear credits toward graduation. We develop an alternative system to place students that uses predictive analytics to combine multiple measures into a placement instrument. Compared to colleges’ existing placement tests, the algorithm is more predictive of future performance. We then conduct an experiment across seven colleges to evaluate the algorithm’s effects on students. Placement rates into college-level courses increased substantially without reducing pass rates. Adjusting for multiple testing, algorithmic placement generally, though not always, narrowed gaps in college placement rates and remedial course taking across demographic groups. A detailed cost analysis shows that the algorithmic placement system is socially efficient: it saves costs for students while increasing college credits earned, which more than offsets increased costs for colleges. Costs could be reduced with improved data digitization, as opposed to entering data by hand.
External Link(s)

Registration Citation

Citation
BERGMAN, PETER, Elizabeth Kopko and Julio Rodriguez. 2021. "Using Predictive Analytics to Track Students: Evidence from a Seven-College Experiment." AEA RCT Registry. June 15. https://doi.org/10.1257/rct.7827-1.0
Experimental Details

Interventions

Intervention(s)
See paper: http://www.columbia.edu/~psb2101/bergmananalytics.pdf
Intervention Start Date
2016-08-01
Intervention End Date
2018-12-31

Primary Outcomes

Primary Outcomes (end points)
See paper.
Primary Outcomes (explanation)
See paper.

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
See paper: http://www.columbia.edu/~psb2101/bergmananalytics.pdf
Experimental Design Details
Randomization Method
Computer
Randomization Unit
Individual
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
n/a
Sample size: planned number of observations
12,544
Sample size (or number of clusters) by treatment arms
6,141 control 6,403 treatment
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
2 percent point increase in course pass rates.
IRB

Institutional Review Boards (IRBs)

IRB Name
Teachers College, Columbia University
IRB Approval Date
2016-04-08
IRB Approval Number
14-361

Post-Trial

Post Trial Information

Study Withdrawal

There are documents in this trial unavailable to the public. Use the button below to request access to this information.

Request Information

Intervention

Is the intervention completed?
Yes
Intervention Completion Date
December 01, 2018, 12:00 +00:00
Data Collection Complete
No
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials