Secondary Outcomes (explanation)
We plan to use detailed chat protocols to examine heterogeneous effects by type of the AI tutor usage. We will operationalize the usage type through several components:
1. Frequency of Usage: The number of sessions during which a student interacts with the Chatbot, measured as unique login events within the study period.
2. Extent of Usage: The cumulative length of interactions, quantified by the total number of words or characters exchanged in the chat sessions.
3. Intent of Chats: We will employ large language models (LLMs) to classify the intent behind student interactions with the Chatbot, e.g. usage for definitions or clarifications, generation of real-world examples, or more general inquiries or other purposes
4. Sophistication: We further employ LLMs to assess the sophistication of prompts regarding complexity, specificity, etc.
5. Interactiveness: We assess the interactiveness of chats by the number of follow-up prompts within a single session, indicating iterative engagement with the Chatbot.