Humans versus Smartbots: Scaling-up behavioral interventions to reduce teacher shortages

Last registered on December 05, 2024

View Trial History

Pre-Trial

Trial Information

General Information

Title

Humans versus Smartbots: Scaling-up behavioral interventions to reduce teacher shortages

RCT ID

AEARCTR-0014942

Initial registration date

December 02, 2024

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published

December 05, 2024, 11:00 AM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Country

Chile

Region

Entire country

Primary Investigator

Name

Nicolas Ajzenman

Affiliation

Sao Paulo School of Economics - FGV

Contact Primary Investigator

Other Primary Investigator(s)

PI Name

Gregory Elacqua

PI Affiliation

IADB

Contact Investigator

PI Name

Catalina Hermosilla

PI Affiliation

IADB

Contact Investigator

PI Name

Graciela Perez Nuñez

PI Affiliation

Unesco

Contact Investigator

PI Name

Ana Teresa del Toro Mijares

PI Affiliation

IADB

Contact Investigator

Additional Trial Information

Status

On going

Start date

2024-11-01

End date

2025-03-31

Keywords

Education

Additional Keywords

Teachers, teacher policy, teacher shortages, scale-up, behavioral, chatbots

JEL code(s)

D91, I23, I25

Secondary IDs

Prior work

This trial does not extend or rely on any prior RCTs.

Abstract

We design an experiment to assess the relative effectiveness of human mentors versus AI-powered chatbots in motivating students to pursue a teaching career in Chile. We work with Elige Educar, a Chilean NGO that implements a mentoring program for students interested in education or related fields. Participating high school students are randomly assigned to one of three groups: a group that re- ceives support from a human mentor via WhatsApp; a group that receives support from an AI-powered chatbot, also through WhatsApp; and a passive control group that receives no support. We assess the effectiveness of each treatment in motivating students to register for an education major through Chile’s centralized university admissions process. Beyond addressing Chile’s immediate teacher recruitment challenges, the study contributes to the emerging research on AI in education by empirically assessing the potential of generative AI to provide personalized, scalable mentoring.

External Link(s)

Registration Citation

Citation

Ajzenman, Nicolas et al. 2024. "Humans versus Smartbots: Scaling-up behavioral interventions to reduce teacher shortages." AEA RCT Registry. December 05. https://doi.org/10.1257/rct.14942-1.0

Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Experimental Details

Interventions

Intervention(s)

The intervention is a randomized controlled trial comparing two methods of mentoring high school students in Chile about teaching careers: human mentors and an AI-powered chatbot named "Kai." Working with the NGO Elige Educar, the study targets students who registered for Chile's university entrance exam (PAES) and expressed interest in education or social sciences careers.

Students are randomly assigned to receive WhatsApp-based mentoring from either trained human mentors (typically education students), an AI chatbot trained on Elige Educar's resources using ChatGPT-4mini, or no support (control group). The mentoring occurs from October 2024 to January 2025, with weekly activation of student subgroups. Both human mentors and the chatbot provide information about education programs, scholarships, career benefits, and application requirements. They make up to two contact attempts per student during their assigned intervention week.

A key innovation is expanding beyond students already interested in teaching to include those interested in social sciences more broadly. The intervention includes an intensive support phase during university applications in January 2025, when both treatment arms contact all non-opted-out students to provide application guidance. The study will evaluate effects on students' likelihood of applying to education programs through Chile's centralized university admissions process.

See PaP for more details.

Intervention (Hidden)

Intervention Start Date

2024-11-01

Intervention End Date

2025-01-31

Primary Outcomes

Primary Outcomes (end points)

The study has three primary outcomes measuring students' preferences for teaching careers:

Whether a student ranks an education major as their first choice in the centralized university application system (main outcome)

Proportion of education majors in a student's choice set

Whether they applied to at least one education major

Data for these outcomes comes from administrative records from DEMRE, Chile's university admissions authority.

Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)

There is also one secondary outcome: actual enrollment in education programs. However, this outcome requires cautious interpretation since enrollment depends on a centralized assignment algorithm that considers all students' choices, meaning treatment effects could be biased by spillovers between treatment and control groups.

Data for these outcomes comes from administrative records from DEMRE, Chile's university admissions authority.

Secondary Outcomes (explanation)

Experimental Design

The study is a randomized controlled trial evaluating the effectiveness of AI-powered chatbots versus human mentors in motivating Chilean high school students to pursue teaching careers. The experiment has three arms:

Human mentor treatment: Students receive WhatsApp mentoring from trained education students using Elige Educar's "Quiero Ser Profe" methodology.
AI chatbot treatment: Students interact with "Kai," an AI-powered chatbot trained on Elige Educar's resources using ChatGPT-4mini.
Control group: No contact or support.

The sample includes 43,539 students who registered for Chile's university entrance exam and expressed interest in education (54.4%) or social sciences (45.6%). Treatment assignment is stratified by gender, school type (public/private-subsidized/private), and initial career interests. The human mentoring arm includes 9,000 students, chatbot arm 20,000 students, and control group 14,539 students.
Both treatment arms operate October 2024-January 2025, making two contact attempts per student during their assigned week. An intensive support phase occurs during university applications (January 2024). The intervention uses Elige Educar's WhatsApp Business account to ensure source credibility.
Primary outcomes measure students' likelihood of applying to education programs through Chile's centralized university admissions system. Administrative data comes from DEMRE (Department of Evaluation, Measurement and Educational Registry).

Experimental Design Details

Randomization Method

Randomization done in office by a computer

Randomization Unit

Student, no clustering.

Was the treatment clustered?

Experiment Characteristics

Sample size: planned number of clusters

No clusters

Sample size: planned number of observations

Total sample size: 43,539 students Breakdown by arm: Human mentor treatment: 9,000 students AI chatbot treatment: 20,000 students Control group: 14,539 students Unit of randomization is individual students (no clustering).

Sample size (or number of clusters) by treatment arms

Total sample size: 43,539 students
Breakdown by arm:

Human mentor treatment: 9,000 students
AI chatbot treatment: 20,000 students
Control group: 14,539 students

Unit of randomization is individual students (no clustering).

Minimum detectable effect size for main outcomes (accounting for sample design and clustering)

Based on Ajzenman et al. (2024) results and power calculations with 0.8 power at 5% significance: Human mentor vs control (n=9,000 vs 14,500): Can detect 1.25 percentage point effect (baseline: 13%) Chatbot vs control (n=20,000 vs 14,500): Can detect 1.0 percentage point effect Human vs chatbot (n=9,000 vs 20,000): Can detect 1.3 percentage point difference Calculations assume no clustering and include demographic controls. Ajzenman, N., Elacqua, G., Jaimovich, A. and Pérez-Núñez, G. (2024). Humans versus chatbots: Scaling-up behavioral interventions to reduce teacher shortages, Journal of Political Economy Microeconomics . Forthcoming.

Supporting Documents and Materials

IRB