Let’s Chat: Leveraging Chatbot Outreach for Improved Course Performance

All students enrolled in the focal courses received standard communications from the course instructor and teaching assistant, as described above. Via the chatbot platform, each course instructional team sent treatment students 2-3 scheduled text messages each week (for a total of about 40 messages throughout the semester). These text messages were designed to: (1) provide timely reminders about course requirements; (2) provide customized feedback on each student’s individual progress; (3) connect students to course-relevant academic supports; and (4) serve as an additional channel of communication between students and their course instructors. The chatbot messages fell into three broad categories: weekly updates, encouragement messages, and reminder messages. Students received weekly updates every Monday to preview their course tasks and responsibilities for that week. These updates were customized by whether students had completed the previous week’s assignments. Encouragement messages were signed by the course TA and were crafted to promote a growth mindset and to invite students to provide feedback on how their semester was going. These encouragement messages were used more frequently in Government. Finally, reminder messages were sent to students as needed (e.g., outreach to students who had not completed an online exam by a given time). When students texted in with a question, the system’s artificial intelligence (AI) responded with the closest match response in the system knowledge base. When the system flagged a response with a low probability of response match, the question was then directed to and answered by the teaching assistant to provide personalized follow-up, as needed. The responses provided by the teaching assistant were then used to update the system knowledgebase. ,
One additional novel feature of the Government course chatbot was a function called #quizme through which students could request a quiz on the course material covered in an upcoming exam. Through #quizme, students could receive and answer a set of multiple-choice questions. For each one, the bot would indicate whether the student answered correctly and/or direct the student to where in the textbook they could read more about the topic and find the correct answer. The bot promoted #quizme in several weekly digests and additional promotional messages. Students could activate #quizme during the two weeks prior to each course exam due date. Since the Microeconomics course did not have exams, GSU did not develop and deploy a #quizme tool for that course.

2021-08-01

2023-05-31

Most outcomes come from deidentified course gradebooks, course learning management system records (e.g., student time spent reading the online American Government textbook), and GSU administrative records, provided directly to the research team for analysis. We also aimed to understand the effect of chatbot communication on students’ class experiences and perceptions of the instructor. To do so, in Government we added questions in the following domains to an existing end-of-course survey that was directed to students in both experimental conditions: organizational support, self-efficacy, and belonging (adapted from PERTS Ascend and Elevate surveys, see Boucher et al., 2021; Paunesku & Farrington, 2020), instructor expectation (adapted from Smith, 2020), perception of achievable challenge (adapted from Mendes et al., 2007), and novel adaptive expectation scenario items developed for the current study. Appendix C reports the specific attitudinal questions we asked students.
Finally, we included a set of survey items to ask treatment participants specifically about their experience with the course chatbot, including the extent to which they found the communication helpful, whether they read the text messages, whether they knew about and/or used the #quizme function (where applicable), and whether they would recommend future use of the chatbot in this and other GSU courses. As we detail below, two limitations to our survey analysis are low response rates and differential survey participation by student characteristics. Other measures of engagement come from the Mainstay message logs. We code incoming student text messages to identify whether and how frequently students messaged the platform as well as characteristics of their messages (e.g., opt-outs vs. questions).

Each semester of the RCT, we identified all students enrolled in the focal courses who had consented to receive text messages from GSU. We randomized these students to either the academic chatbot treatment condition or to the control group. We separately randomized students enrolled as of the first day of class and a second roster of students who enrolled during the semester add/drop period. As a result of these enrollment patterns, students in the first round of randomization received an additional week of messaging (a welcome message and note about first week assignments) relative to students in the second round of randomization. We account for randomization blocks in our analysis (see analytic model below). We do not remove students from analysis who dropped the course during the add/drop period since dropping the course occurred after treatment began.

Randomization done in office by a computer

Student

No

2483 students

2483 students

Approximately 1241 treated and 1241 control students

0.157