Humans versus Smartbots: Scaling-up behavioral interventions to reduce teacher shortages

The intervention is a randomized controlled trial comparing two methods of mentoring high school students in Chile about teaching careers: human mentors and an AI-powered chatbot named "Kai." Working with the NGO Elige Educar, the study targets students who registered for Chile's university entrance exam (PAES) and expressed interest in education or social sciences careers.

Students are randomly assigned to receive WhatsApp-based mentoring from either trained human mentors (typically education students), an AI chatbot trained on Elige Educar's resources using ChatGPT-4mini, or no support (control group). The mentoring occurs from October 2024 to January 2025, with weekly activation of student subgroups. Both human mentors and the chatbot provide information about education programs, scholarships, career benefits, and application requirements. They make up to two contact attempts per student during their assigned intervention week.

A key innovation is expanding beyond students already interested in teaching to include those interested in social sciences more broadly. The intervention includes an intensive support phase during university applications in January 2025, when both treatment arms contact all non-opted-out students to provide application guidance. The study will evaluate effects on students' likelihood of applying to education programs through Chile's centralized university admissions process.

See PaP for more details.

2024-11-01

2025-01-31

The study has three primary outcomes measuring students' preferences for teaching careers:

Whether a student ranks an education major as their first choice in the centralized university application system (main outcome)

Proportion of education majors in a student's choice set

Whether they applied to at least one education major

Data for these outcomes comes from administrative records from DEMRE, Chile's university admissions authority.

There is also one secondary outcome: actual enrollment in education programs. However, this outcome requires cautious interpretation since enrollment depends on a centralized assignment algorithm that considers all students' choices, meaning treatment effects could be biased by spillovers between treatment and control groups.

Data for these outcomes comes from administrative records from DEMRE, Chile's university admissions authority.

The study is a randomized controlled trial evaluating the effectiveness of AI-powered chatbots versus human mentors in motivating Chilean high school students to pursue teaching careers. The experiment has three arms:

Human mentor treatment: Students receive WhatsApp mentoring from trained education students using Elige Educar's "Quiero Ser Profe" methodology.
AI chatbot treatment: Students interact with "Kai," an AI-powered chatbot trained on Elige Educar's resources using ChatGPT-4mini.
Control group: No contact or support.

The sample includes 43,539 students who registered for Chile's university entrance exam and expressed interest in education (54.4%) or social sciences (45.6%). Treatment assignment is stratified by gender, school type (public/private-subsidized/private), and initial career interests. The human mentoring arm includes 9,000 students, chatbot arm 20,000 students, and control group 14,539 students.
Both treatment arms operate October 2024-January 2025, making two contact attempts per student during their assigned week. An intensive support phase occurs during university applications (January 2024). The intervention uses Elige Educar's WhatsApp Business account to ensure source credibility.
Primary outcomes measure students' likelihood of applying to education programs through Chile's centralized university admissions system. Administrative data comes from DEMRE (Department of Evaluation, Measurement and Educational Registry).

Not available

Randomization done in office by a computer

Student, no clustering.

No

No clusters

Total sample size: 43,539 students Breakdown by arm: Human mentor treatment: 9,000 students AI chatbot treatment: 20,000 students Control group: 14,539 students Unit of randomization is individual students (no clustering).

Total sample size: 43,539 students
Breakdown by arm:

Human mentor treatment: 9,000 students
AI chatbot treatment: 20,000 students
Control group: 14,539 students

Unit of randomization is individual students (no clustering).

Based on Ajzenman et al. (2024) results and power calculations with 0.8 power at 5% significance: Human mentor vs control (n=9,000 vs 14,500): Can detect 1.25 percentage point effect (baseline: 13%) Chatbot vs control (n=20,000 vs 14,500): Can detect 1.0 percentage point effect Human vs chatbot (n=9,000 vs 20,000): Can detect 1.3 percentage point difference Calculations assume no clustering and include demographic controls. Ajzenman, N., Elacqua, G., Jaimovich, A. and Pérez-Núñez, G. (2024). Humans versus chatbots: Scaling-up behavioral interventions to reduce teacher shortages, Journal of Political Economy Microeconomics . Forthcoming.