Intervention(s)
The study was conducted in a free, online 4-week long online peer SAT math tutoring bootcamp on the Schoolhouse.world platform. Anyone with an SAT subject score of 650 or above could apply to serve as a peer tutor for teaching that subject. As long as they complete the Schoolhouse asynchronous tutor training, they are then eligible to teach their first bootcamp. Our participant sample consists of all instructors and students in the April/May 2024 and the June 2024 bootcamps.
In the May bootcamp, tutors in the treatment arms of the RCT received an email prior to the start of the bootcamp informing them they would receive feedback and explaining the relevant parts of the feedback modal. For the June bootcamp, tutors were not primed to receive feedback.
Feedback to Instructors
Instructors in the TutorFeedback and TutorStudentFeedback conditions received automated feedback with the following components:
Introduction to the feedback
Summary statistics of the session and comparison to the previous session
tutor talk percentage
proportion of students engaged
Description of talk move in focus for that session
Their talk moves in action (list of talk moves from their transcript)
For first two sessions: link to relevant training module
GPT-4 Turbo generated actionable suggestions for next session
Reflection opportunity
The talk moves that were in focus changed over the course of the bootcamp, following a pre-defined curriculum of talk moves for each of the 8 sessions:
Session #
1: Eliciting ideas from students
344/363 tutors in the treatment conditions did not receive feedback for session 1 during June bootcamp due to an error
2: Eliciting ideas from students
3: Revoicing student ideas
4: Revoicing student ideas
5: No feedback
6: Prompting for reasoning
7: Prompting for reasoning
8: No feedback - received end-of-bootcamp survey on the AI feedback
The talk moves are defined as:
Inviting learner ideas
This talk move was identified by first filtering session transcripts with a fine-tuned question detection model, which isolates utterances resembling questions. The questions are then passed to an Electra-base model, fine-tuned to identify the “pressing for reasoning” and “pressing for accuracy” labels from the TalkMoves Dataset by Suresh et al. (2022)
Building on learner ideas
For identifying this talk move, we used the uptake model developed by Demszky et al. (2021). This model analyzes utterances from the session transcripts to pinpoint when tutors effectively engage with and extend student contributions.
Pressing for reasoning
For the sessions with this focus, the same Eliciting model was used as the “inviting learner ideas” sessions.
After a tutor taught their section on Zoom, their transcript was analyzed through our automated analysis pipeline. The analysis was usually completed within a few hours. Once the feedback for the most recent session becomes available, tutors receive an email notifying and encouraging them to log into the Schoolhouse.world platform to view it. The feedback appears as a pop-up modal the next time they log into Schoolhouse.world. All previous feedback can be accessed again from the tutor’s personal profile page.
Edge-cases: if a tutor misses a session, they do not receive any feedback. They will receive feedback after the next session they teach. Tutors did not substitute teaching other cohorts.
All instructors were required to complete training about Schoolhouse’s MARS rubric (Mastery, Active Learning, Respectful Community, and Safety), general SAT knowledge, and new information about the digital SAT. They also participate in a 1-hour live onboarding session.
Feedback to Students
Students in the TutorStudentFeedback groups received automated feedback on their engagement in the tutoring session. The feedback included the following components:
Student talk time ratio in section
Motivational message to encourage students to participate in the session
Students were randomized to receive one of two types of motivational messages. Students in the TutorStudentFeedbackSelf group received a message that encouraged them to participate in order to optimize their own learning:
Students in the TutorStudentFeedbackSocial group received a message that encouraged them to participate in order to help everyone else learn:
Students received the feedback from their previous session as a pop-up modal right before they join their next session.
If a student did not attend their section, they did not receive feedback. If a student attended a session of a different tutor, they received the treatment condition assigned to that new tutor (i.e. if they were in control but then dropped-in on another session in the treatment group, they received feedback for that session) -- this did not happen often, if at all.
At the end of the study
After the last session, automated feedback to tutors included a few survey questions to probe their perceptions of the automated feedback:
We also interviewed a sample of tutors and students to gauge their perception of the feedback. A random sample of tutors and students from each treatment arm of the study was emailed after the Bootcamp with an invitation to sign up for an interview in exchange for a $15 Amazon gift card. In total, 17 tutors were interviewed, with 5, 6, and 6 from each arm; 9 learners were interviewed, with 3, 2, and 4 from each arm. The interviews were conducted virtually over Zoom by a member of the Schoolhouse team. In the first phase of the interview, the interviewee was asked about their overall experience in the SAT Bootcamp; in the second phase, the interviewee was shown their automated feedback and asked a set of questions about how they felt about it. Finally, the interviews with tutors were different from the interviews with students; each was tailored to the specifics of the feedback they had received and their role in the Bootcamp.