Building Expertise in College Students in the Era of Generative AI: A Randomized Evaluation at Four Engineering Colleges in India

Students in the course group take a programming course over one academic year, combining in-person and online sessions. The course teaches programming alongside how to use AI tools and how to check their work. Students in the comparison group attend a small number of online seminars on unrelated topics. Both groups continue their regular college classes.

2026-10-04

2027-07-31

1. Programming fundamentals (endline tasks completed without generative AI)
2. Evaluative judgment (diagnosis, evaluation, and correction of problems in code and analyses)
3. Ability to direct generative AI (endline task completed with generative AI)
4. Quality of work completed with generative AI
5. Placement: received a campus or verified off-campus job offer by graduation
6. Starting salary: annual offered compensation, with zero for students without an offer
7. Job type: offer is for a software or technical role

Outcomes 1 to 4 come from a screen-recorded endline assessment in September and October 2027, administered in college computer laboratories on new parallel forms. Programming fundamentals is scored from code written, interpreted, and modified without generative AI: whether it runs and meets the task specification, on a data-analysis task and on a common item in which every student evaluates and repairs the same unfamiliar code. Evaluative judgment is scored as three separately credited accomplishments: identifying and explaining a problem before the generative AI tool raises it, judging whether a claim or solution is justified (which includes correctly accepting correct output), and making or directing an adequate correction. Direction of generative AI is scored from the tool transcripts: whether the student communicated the relevant requirements and context, recognized when more instruction was needed, and guided an adequate revision. Pasting the task whole is not penalized in itself. Quality of work completed with generative AI is scored against each form's answer key as the correctness of the conclusion and of the data preparation. Graders work from a written codebook, alongside a generative AI grader. A random subsample is double-scored, and graders do not know students' assignment. Each component is standardized against the control group's endline mean and standard deviation. Indices are the equally weighted mean of standardized components.

Outcomes 5 to 7 come from SVES placement records for the placement season of November 2027 to May 2028, through graduation in mid-2028. Salary is measured in rupees per year (cost to company), winsorized at the 99th percentile, and set to zero for students without an offer. The version conditional on having an offer is reported as secondary. Job type is coded from the job title and employer using a classification fixed before the placement data are opened.

Calibration (signed and absolute difference between the student's self-rating and a grader's rating); recall of one's own conclusion; verification behavior (concrete checks, reading generated code); the order in which students first identify problems; how students report using generative AI (frequency, first resort when stuck, substituting for versus supporting their own work, paying for tools); programming self-efficacy; semester grades and backlogs; conditional starting salary and number of offers; employment and job type from public professional profiles of consenting graduates after graduation.

Calibration, recall, and verification come from the reflection that students complete without generative AI after the assisted task, checked against the screen recording. Self-reports come from the endline survey, which repeats the baseline questions on use. Grades come from college records for consenting students. Post-graduation outcomes are exploratory, because they depend on separately consented public profiles.

Individual-level randomized trial. Students who took part in a baseline survey and assessment are assigned by lottery to the course or the comparison group, and outcomes are compared between the two groups.

Not available

Randomization done in office by a computer

Individual student

No

0

3500

1,200 training (300 per campus); about 1,900 to 2,300 control

With 1,200 training and 1,923 control students, a two-sided 5 percent test with 80 percent power detects 0.086 SD if the baseline score explains 30 percent of the variance in the endline score. With 20 percent attrition it detects 0.096 SD, or 0.115 SD with no gain in precision from covariates. For placement, a binary outcome available from college records for the full sample, the MDE is 4.7 to 5.2 percentage points for placement rates between 30 and 70 percent.