THE EFFECTS OF PERSONALIZED FEEDBACK: Replication in a non-pandemic context.

Last registered on August 24, 2023
View Trial History

Pre-Trial

Trial Information

General Information

Title
THE EFFECTS OF PERSONALIZED FEEDBACK: Replication in a non-pandemic context.
RCT ID
AEARCTR-0011939
Initial registration date
August 17, 2023

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
August 24, 2023, 4:58 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Primary Investigator

Name
Esteban Aucejo
Affiliation
Arizona State University
Contact Primary Investigator

Other Primary Investigator(s)

PI Name
Kelvin Wong
PI Affiliation
Arizona State University
Contact Investigator

Additional Trial Information

Status
In development
Start date
2023-08-23
End date
2024-08-01
Keywords
Education
Additional Keywords
JEL code(s)
I2
Secondary IDs
Prior work
This trial is based on or builds upon one or more prior RCTs.
Abstract
This study examines the effects of scalable, personalized email feedback on students at a large public institution in the US. We aim to discern which feedback types most significantly influence student learning and retention, specifically identifying the content that most likely alters student behavior. Our research closely mirrors the experiment presented in the working paper “The Effect of Feedback on Student Performance” by Esteban Aucejo and Kelvin Wong.

Our analysis seeks to validate these findings in an environment where the pandemic does not influence classes (Fall 2023). The pandemic led to diminished interactions between students and professors due to the reliance on platforms like Zoom for instruction. In such a scenario, personalized feedback emails might have held greater significance for students, potentially resulting in more pronounced effects.

We aim to evaluate the varied effects of feedback messages on three distinct student groups:

a) First-generation students (those whose parents did not attend college) versus non-first-generation students,
b) Over-optimistic students (defined by the disparity between their expected class rank and their rank after the first exam) versus their non-overoptimistic counterparts,
c) Mode of instruction students received: traditional in-person, hybrid, or online classes.

To gauge the influence of feedback, we will evaluate both educational outcomes and qualitative student responses. We will explore students' perceptions of personalized feedback messages and determine how their beliefs shift post-feedback.
External Link(s)

Registration Citation

Citation
Aucejo, Esteban and Kelvin Wong. 2023. "THE EFFECTS OF PERSONALIZED FEEDBACK: Replication in a non-pandemic context.." AEA RCT Registry. August 24. https://doi.org/10.1257/rct.11939-1.0
Sponsors & Partners
Experimental Details

Interventions

Intervention(s)
Students enrolled in introductory economics courses will receive personalized feedback emails based on their performance in the initial class exam. The distribution of these emails will be randomized yet stratified by factors such as first-generation status, gender, and letter grade achieved in the first exam. Additionally, we will conduct two surveys to collect background data and gauge any shifts in students' beliefs regarding their capabilities.
Intervention Start Date
2023-10-01
Intervention End Date
2023-12-22

Primary Outcomes

Primary Outcomes (end points)
We intend to evaluate the following primary outcomes:
1. Performance in the second exam,
2. Final course grade,
3. Performance in the second exam and final grade in courses where no exams are dropped (given that some courses permit students to exclude an exam score),
4. Percentage of problem sets successfully completed (i.e., passed),
5. Whether the student opts to change their major or field of study, and
6. The student's perception of the usefulness of specific types of feedback.
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
Students from a largepublic higher education institution, specifically those enrolled in economics courses, will receive varied feedback based on their exam results. The objective is to ascertain the impact of personalized feedback on student outcomes.
Experimental Design Details
Not available
Randomization Method
The randomization will be done in an office by a computer using the Stata command "stratarand".
Randomization Unit
Randomization will be done at the student level, conditional on attending the same class (we have approximately 20 to 23 classes in our experiment). Students will receive a feedback email at random conditional on their performance in the class and providing consent in the initial survey. Treatment assignment will be done by stratifying the sample.
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
No clusters, we will randomize students into the treatment and control groups within the classroom. We plan to perform the randomization on approximately 20 to 23 classrooms. The final number of classrooms still depends on receiving consent from a couple of professors in charge of a few classrooms.
Sample size: planned number of observations
Our sample involves approximately 2000 students divided across different classrooms. The final number of students will depend on the share of them providing consent to participate in the program. Randomization will be performed on the sample of students that provided consent conditional on classroom.
Sample size (or number of clusters) by treatment arms
Approximately half students will be randomly allocated to the treatment groups and the other half to the control group.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Supporting Documents and Materials
IRB

Institutional Review Boards (IRBs)

IRB Name
Arizona State University
IRB Approval Date
2023-08-03
IRB Approval Number
STUDY00013095 (The original IRB was recently updated to reflect the new sample)
Analysis Plan