Can Artificial Intelligence Improve Writing? Experimental Evidence on AWE-Based ed-techs

Last registered on December 30, 2018

View Trial History

Pre-Trial

Trial Information

General Information

Title

The Impacts of Automated Essay Scoring on Writing Skills and Access to College

RCT ID

AEARCTR-0003729

Initial registration date

December 28, 2018

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published

December 30, 2018, 10:03 PM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Country

Brazil

Region

Espírito Santo

Primary Investigator

Name

Flávio Riva

Affiliation

Sao Paulo School of Business Administration

Contact Primary Investigator

Other Primary Investigator(s)

PI Name

Lycia Lima

PI Affiliation

Sao Paulo School of Business Administration

Contact Investigator

PI Name

Bruno Ferman

PI Affiliation

Sao Paulo School of Economics

Contact Investigator

Additional Trial Information

Status

In development

Start date

2019-03-01

End date

2019-11-30

Keywords

Education

Additional Keywords

writing skills, ed-techs, automated essay scoring

JEL code(s)

Secondary IDs

Abstract

Artificial intelligence has the potential of – perfectly or imperfectly – substituting and/or complementing time-intensive teacher tasks. However, evidence on whether and how this impacts learning is still scant. We propose a randomized evaluation of a program that uses automated essay scoring to provide personalized feedback on writing skills to students. Teachers also receive a summary of error patterns, which may help them tailor class content to students’ specific needs. The current structure of the program – which integrates automated scoring and human graders – motivated the evaluation design, which will test not only the impact of the algorithm but also the marginal effect of human graders, a relevant feature for scalability. The results will shed light on the question of whether and how this ed-tech can be used to improve learning in a low-quality (public schools) post-primary education context.

External Link(s)

Registration Citation

Citation

Ferman, Bruno, Lycia Lima and Flávio Riva. 2018. "The Impacts of Automated Essay Scoring on Writing Skills and Access to College." AEA RCT Registry. December 30. https://doi.org/10.1257/rct.3729-1.0

Former Citation

Ferman, Bruno, Lycia Lima and Flávio Riva. 2018. "The Impacts of Automated Essay Scoring on Writing Skills and Access to College." AEA RCT Registry. December 30. https://www.socialscienceregistry.org/trials/3729/history/39694

Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Experimental Details

Interventions

Intervention(s)

There are two intervention arms: (i) providing a school access to an online platform with an automated essay scoring (AES) algorithm that gives feedback on a broad set of writing features for seniors in public school secondary students in Brazil; (ii) providing the school with the same platform, but automatically corrected essays are assigned to professional human graders independently hired, who validate the corrections/feedback from the algorithm and typically include more personalized comments on the essay content, structure and on ways to improve it. In both cases, all the information generated by the artificial and human intelligence is available to teachers.

Intervention Start Date

2019-03-01

Intervention End Date

2019-11-30

Primary Outcomes

Primary Outcomes (end points)

The key outcome variables are: (a) enrollment and participation in the National Secondary Education Exam (“Exame Nacional do Ensino Medio”, ENEM), a non-compulsory standardized test that has been used by a large number of post-secondary institutions for admission purposes; (b) achievement in the argumentative essay (and other parts, which are standardized test scores in Mathematics, Language and Natural Sciences) of ENEM; (c) students’ proficiency in language and mathematics, using administrative data from the state’s standardized test; (d) writing skills, captured by independently hired graders, who will grade small writing prompts both in terms of form (handwriting, spelling, punctuation, sentence structure/grammar) and content (vocabulary, organization/overall structure and ideas); (d) admission and progress in postsecondary education; (e) future labor market outcomes, like formality and work earnings

Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)

Secondary Outcomes (explanation)

Experimental Design

The evaluation design will be based on the random allocation of a sample of public schools with at least 5 computers in Espírito Santo state into one of the three conditions for the 2019 academic year: (i) control; (ii) platform with the AES algorithm and (iii) platform with the AES algorithm and human essay graders. More details on the interventions can be found above.

Experimental Design Details

Randomization Method

Randomization done in office by a computer

Randomization Unit

School

Was the treatment clustered?

Yes

Experiment Characteristics

Sample size: planned number of clusters

150 schools

Sample size: planned number of observations

15,000 students

Sample size (or number of clusters) by treatment arms

50 schools control, 50 schools to the treatment that gives access to platform with the AES algorithm, 50 schools to the additional treatment involving human graders

Minimum detectable effect size for main outcomes (accounting for sample design and clustering)

Based on simulations using administrative data from state’s 2017 standardized language test scores, for a significance level of 0.05 and power of 0.8, we expect a minimum detectable effect (MDE) of around 0.1 standard deviation for the effects of each treatment, once we control for individual-level pre-intervention test scores. We also reached a minimum detectable effect (MDE) of 0.1 standard deviation for the comparison between the two treatments, using the same simulations.

Supporting Documents and Materials

IRB