The StudyChat Dataset: Student Dialogues With ChatGPT in an Artificial Intelligence Course
This provides a dataset for researchers to study LLM impacts in education, but it is incremental as it focuses on data collection and analysis without new methods.
The researchers tackled the problem of understanding student usage patterns of LLM-powered tutoring tools in education by introducing the StudyChat dataset, which captures 16,851 real-world student interactions in an AI course, and found that students prompting for conceptual understanding and coding help performed better, while those using LLMs to circumvent assignments had lower exam outcomes.
The widespread availability of large language models (LLMs), such as ChatGPT, has significantly impacted education, raising both opportunities and challenges. Students can frequently interact with LLM-powered, interactive learning tools, but their usage patterns need to be monitored and understood. We introduce StudyChat, a publicly available dataset capturing real-world student interactions with an LLM-powered tutoring chatbot in a semester-long, university-level artificial intelligence (AI) course. We deploy a web application that replicates ChatGPTs core functionalities, and use it to log student interactions with the LLM while working on programming assignments. We collect 16,851 interactions, which we annotate using a dialogue act labeling schema inspired by observed interaction patterns and prior research. We analyze these interactions, highlight usage trends, and analyze how specific student behavior correlates with their course outcome. We find that students who prompt LLMs for conceptual understanding and coding help tend to perform better on assignments and exams. Moreover, students who use LLMs to write reports and circumvent assignment learning objectives have lower outcomes on exams than others. StudyChat serves as a shared resource to facilitate further research on the evolving role of LLMs in education.