The Effect of Giving Instructors Agency to Choose Among Different Types of Automated, Natural Language Processing Based Feedback

Last registered on December 21, 2023
View Trial History

Pre-Trial

Trial Information

General Information

Title
The Effect of Giving Instructors Agency to Choose Among Different Types of Automated, Natural Language Processing Based Feedback
RCT ID
AEARCTR-0012746
Initial registration date
December 21, 2023

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
December 21, 2023, 8:07 AM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Country
United States of America
Region

Primary Investigator

Name
Dora Demszky
Affiliation
Stanford University
Contact Primary Investigator

Other Primary Investigator(s)

PI Name
Heather Hill
PI Affiliation
Harvard University
Contact Investigator
PI Name
Eric Taylor
PI Affiliation
Harvard University
Contact Investigator
PI Name
Chris Piech
PI Affiliation
Stanford University
Contact Investigator

Additional Trial Information

Status
Completed
Start date
2023-04-01
End date
2023-06-22
Keywords
Education
Additional Keywords
natural language processing, automated feedback, online course
JEL code(s)
Secondary IDs
Prior work
This trial is based on or builds upon one or more prior RCTs.
Abstract
This project builds on a previous study conducted in 2021 on Code in Place, a five-week-long online programming course where we found that automated feedback to instructors can improve their instruction and student satisfaction. The current study was conducted in the spring of 2023 on Code in Place, and its goal is to understand whether providing instructors agency in choosing the type of automated feedback they would like to receive influences their engagement with and impact of the feedback. Learner agency is thought to enhance engagement and improve outcomes, but few empirical studies have examined its effect in instructor learning settings. To answer this question, the study leverages both manual annotation and computational natural language processing techniques.
External Link(s)

Registration Citation

Citation
Demszky, Dora et al. 2023. "The Effect of Giving Instructors Agency to Choose Among Different Types of Automated, Natural Language Processing Based Feedback." AEA RCT Registry. December 21. https://doi.org/10.1257/rct.12746-1.0
Sponsors & Partners
Experimental Details

Interventions

Intervention(s)
Please refer to Analysis Plan document.
Intervention (Hidden)
Intervention Start Date
2023-04-20
Intervention End Date
2023-06-10

Primary Outcomes

Primary Outcomes (end points)
RQ1: Engagement with feedback
RQ2: Perception of feedback
RQ3: Instructional practices
RQ4: Student outcomes
Please refer to Analysis plan for further details.
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
Sample size: 588 instructors
The study was conducted in a free, online 6 week long introductory programming course called Code in Place. Anyone could apply to serve as a volunteer section leader on the course, and then they were selected by the course organizers. Our participant sample consists of all adult (18+) instructors in Code in Place.

Before the course began: We randomized instructors once they were accepted to teach in the course, and before the course began. Half of the instructors got a choice for what type of automated feedback they wanted to receive (see figure below). Instructors were asked to make this choice on the Code in Place website, and this action item was listed on their pre-course checklist. The choice involved feedback on 3 types of talk moves (Getting ideas on the table, Building on student ideas, Orienting students to one another), which they could select for pairs of weeks (1-2, 3-4, weeks 5-6*). Instructors also had the option to enable experimental, GPT-4 based feedback for the last 2 weeks and had the option to compare their metrics with other section leaders for all weeks. We displayed a short definition and an example below each talk move to help inform their choices. We sent email nudges to instructors before the course began to make a choice.

*We had thought that the course would only be 5 weeks long, hence the choice interface only had Week 5 listed for the third box; as we realized the course would be 6 weeks long, we applied their choices for week 5 to week 6 as well.



The control group did not get to choose. Instead the control group was randomly assigned to feedback under the constraint that the distribution of feedback patterns in the control group was the same as the distribution in the treatment group. For example, 36% of the treatment group chose the pattern: 2 weeks on getting ideas on the table, 2 weeks on building on student ideas, 2 weeks of experimental feedback. Thus 36% of the control group were assigned to that same pattern. This assignment of the control group means that the only difference between treatment and control, in expectation, is whether the instructor chose their pattern of feedback or were assigned their pattern of feedback.

About 80% of treatment group instructors made a choice of what feedback to receive. The 20% that did not choose got were assigned feedback with a similar method as the control group.

All instructors had access to training modules that explained each talk move and showed animations to illustrate the talk move. Both control and treatment group instructors were encouraged to complete these modules prior to their first section. Completion rates were about 40% for each of the training modules.

Choice of feedback: 36% of instructors choose the following sequence of feedback: Getting ideas on the table, building on student ideas, then the experimental (GPT-4) feedback with the choice to compare their metrics to other section leaders; this sequence of feedback was by far the most popular choice. The rest of the instructors choose a combination of other feedback choices.

Once the course began: After each of their six sessions, instructors received automated feedback based on their chosen or assigned feedback type. The feedback was released to everyone on Saturday (sections taught Wed-Friday) via email. The email varied: the treatment group was reminded that they got a choice and here’s the feedback on that topic. The control group did not get this reminder that they had made a choice. Below is an example email that the treatment group received.

All instructors could view their automated feedback on the Code in Place webpage. The talk move feedback included several components:
Introduction to the feedback
Summary statistics for the given talk move
Definition of each talk move
Their talk moves in action (list of talk moves from their transcript)
Link to the relevant training module
Reflection opportunities
Experimental Design Details
Randomization Method
Hashing function (similar to coin flip)
Randomization Unit
Instructor
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
N/A
Sample size: planned number of observations
588
Sample size (or number of clusters) by treatment arms
50% split between observations
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Supporting Documents and Materials
IRB

Institutional Review Boards (IRBs)

IRB Name
Stanford IRB
IRB Approval Date
2023-01-05
IRB Approval Number
68376
Analysis Plan

Analysis Plan Documents

Study Details & Analysis Plan

MD5: 23374892812d6f78e2bcf9ef0682e161

SHA1: e317129bc2725ed4ec46f43a80a4f494ea611ee0

Uploaded At: December 21, 2023

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials