Impact of introducing AI technology on students' writing skills: evidence for Brazil

Last registered on April 13, 2023
View Trial History

Pre-Trial

Trial Information

General Information

Title
Impact of introducing AI technology on students' writing skills: evidence for Brazil
RCT ID
AEARCTR-0010887
Initial registration date
April 01, 2023

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
April 13, 2023, 3:23 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Country
Brazil
Region
Alagoas's state

Primary Investigator

Name
Rafael Barbosa
Affiliation
Federal University of Ceará
Contact Primary Investigator

Other Primary Investigator(s)

PI Name
Mayra Ponti
PI Affiliation
Centro Universitário Barão de Mauá
Contact Investigator

Additional Trial Information

Status
In development
Start date
2023-03-15
End date
2023-06-30
Keywords
Education
Additional Keywords
Artificial Intelligence; student's writing skills; developing countries
JEL code(s)
I2; I25; I29
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
In this study, we evaluated the impact of the introduction of Artificial Intelligence technology in the correction of Portuguese essays on students' writing skills. To do so, teachers in three municipalities in the state of Alagoas, a poor state in Brazil, were randomly assigned to access an online platform for three months. This platform allows teachers to upload essay texts produced by students, giving back to the teachers the students' performance in four aspects of writing skills: formal writing register, cohesion, thematic coherence, and textual typology. The objective of the experiment is to verify whether there has been an improvement in the quality of textual production of primary school students.

External Link(s)

Registration Citation

Citation
Barbosa, Rafael and Mayra Ponti. 2023. "Impact of introducing AI technology on students' writing skills: evidence for Brazil." AEA RCT Registry. April 13. https://doi.org/10.1257/rct.10887-1.0
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
The intervention will be carried out among schools in three municipalities in the state of Alagoas: Arapiraca and Girau do Ponciano. These municipalities have 43 elementary schools and approximately 3200 students in the 6th grade. Randomization will be carried out using a stratification strategy at the school level. The following variables will be used to stratify the schools: school size, measured by the school average class size, and rural vs. urban. The treatment will consist of access to the teachers of the treated schools to an online platform that allows teachers to upload their students' writing texts and return them back to them analyzed in four aspects: formal writing register, cohesion, thematic coherence, and textual typology. On this platform, besides the feedback on the corrected essays, pedagogical materials will also be available for teachers to use in the classroom.

Before the experiment starts, a baseline will be conducted with students and teachers. This baseline will consist of an essay written by the students on a topic appropriate to their grade level. Additionally, information will be collected on the socio-demographic characteristics of the students. The teachers will answer an online questionnaire that will collect information about their work routine, their self-efficacy in teaching text production, and their ICT skills. To expand the sample of teachers, this online questionnaire will be applied to all elementary school teachers in the participating schools, not only to 6th-grade teachers.

At the end of the intervention, the endline will be applied to both students and teachers. Students will perform a written production exercise similar to the baseline. And teachers will be asked again about their work routine during the intervention, use of feedback, self-efficacy in teaching writing skills, and questions related to the use of available teaching resources.

Intervention Start Date
2023-03-15
Intervention End Date
2023-06-30

Primary Outcomes

Primary Outcomes (end points)
Student's writing skills
Primary Outcomes (explanation)
Students' writing skills are based on four aspects: formal writing register, cohesion, thematic coherence, and textual typology. These aspects are based on a pedagogical correction matrix developed by a partner of the implementer, specialized in elementary education. The student's performance is measured by an artificial intelligence device trained on nearly 700,000 similar essays applied nationwide. Other aspects of the student's writing quality can also be evaluated, such as the number of words written correctly and the use of specific sentences appropriate for school age.

Secondary Outcomes

Secondary Outcomes (end points)
The secondary results refer to the teachers' practices when using the online platform. They will be evaluated:
1. time available for teaching activities (teaching grammar, literature, and developing writing skills)
2. Self-efficacy in the development of students' writing skills
3. Number of writing exercises
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
The experiment will be conducted in three Brazilian municipalities: Arapiraca and Girau do Ponciano

Experimental Design Details
Randomization Method
Randomization will be carried out using a stratification strategy at the school level. The following variables will be used to stratify the schools: performance in IDEB, and rural vs. urban.
Randomization Unit
Strata
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
The sample contains 43 elementary schools with approximately 3200 students and 270 teachers.
Sample size: planned number of observations
1678
Sample size (or number of clusters) by treatment arms
22 schools in treatment, 21 schools control
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Supporting Documents and Materials
IRB

Institutional Review Boards (IRBs)

IRB Name
IRB Approval Date
IRB Approval Number
Analysis Plan

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials