Intervention(s)
This randomized control trial was conducted through the TeachFX platform on 523 teachers from Utah school districts and 21 tutors from an online tutoring company, Cignition. This study involved a weekly email sent to the teachers in the treatment group every Tuesday early morning, with the number of focusing questions they asked in all class recordings in the previous week as well as a display of at most top 3 chosen focusing questions. Focusing questions probe students’ thinking instead of guiding them towards a predetermined answer (Alic et al., 2022).
The email also contains a link to the focusing questions insight page on the TeachFX app and to a resources page (https://medium.com/dorademszky/resources-for-focusing-questions-47bc6cdd9953) explaining what focusing questions are and how to ask more focusing questions to students. Further, it includes the top 3 focusing question starters seen in the week (which are identified across all teachers by the two annotators mentioned above). Below are screenshots of an email that was sent to a teacher.
The treatment group also gets to see the focusing questions they asked on the TeachFX app, while the control group does not have access to this insight.
Detecting focusing questions
The focusing questions are identified by the machine learning model we have built to classify teacher utterances as being instances of focusing questions or not. We have fine-tuned a large language model (BERT) on data from the National Council of Teaching in English (NCTE) - this data has annotations for whether or not a teacher utterance is a focusing question. Given a transcript of a class recording, we extract and pre-process teacher utterances and feed it to this binary classification ML model, which tells us whether or not each utterance is a focusing question. In a given week, for each teacher in the treatment group, we store the list of focusing questions chosen by the model for each recording the teacher made that week.
TeachFX sent this data to a pair of annotators, who label (at most) the top 3 focusing questions for a given teacher, for all teachers in the treatment group who recorded that week. These are the questions we surface in the email.
Raffle
As TeachFX realized that it can be difficult to ensure that teachers record consistently across weeks, TeachFX decided to send raffles to teachers in Utah to incentivize teachers to record more consistently. These raffles involved the opportunity to win Amazon gift cards worth $250. Here are the details for the 4 raffles that TeachFX conducted:
Raffle 1: Any teacher that recorded at least once per week between October 10th, 2022 and November 10th, 2022 would have the opportunity to win a $250 Amazon gift card.
Raffle 2: Any teacher that recorded at least 5 times between December 5th, 2022 and December 23rd, 2022 would have the opportunity to win a $250 Amazon gift card.
Raffle 3: Any teacher that recorded at least once per week between February 6th, 2023 and February 17th, 2023 would have the opportunity to win a $250 Amazon gift card.
Raffle 4: Any teacher that recorded at least once per week between February 27th, 2023 and March 10th, 2023 would have the opportunity to win a $250 Amazon gift card.
Other Emails Encouraging Teachers to Record Consistently:
TeachFX also sent out a ‘Words you used the most’ email at the end of October to all Utah teachers who had recorded that month. For each teacher, this email displayed the most frequent words spoken by the teacher and their students (through teacher and student word clouds for the month) in the past month. It also included a reminder about the October-November raffle time period that would end mid November. It encouraged teachers that they were on track to be considered for the raffle prize if they continued to record every week. This email also demonstrated to teachers the fact that they could receive similar insights if they continued to record consistently.
Post-intervention procedures
After any teacher completes 5 weeks of recordings, we consider the research study complete for the teacher and stop sending emails to that teacher. Following this, TeachFX sent the teacher a survey. This survey has the following questions: Teacher survey for focusing RCT - v2
Further, in order to incentivize teachers to complete the survey, TeachFX frames the survey email as follows: “Thank you for using TeachFX to reflect on your teaching practice! To learn how to better support teachers and improve our app insights, we are sending you a survey about your background and the teaching practices used in your math/science lessons. This survey will take no more than 5 minutes to complete, and your identity will remain strictly confidential. To show our appreciation for your time, we will send you a $10 Starbucks gift card for completing the survey.”
We also plan to conduct interviews with all teachers who complete the survey in order to understand more about their background, their experience with TeachFX as well as (for the treatment group), the impact of getting feedback on focusing questions on their teaching.