Thematic and Task-Based Categorization of K-12 GenAI Usages with Hierarchical Topic Modeling
This work addresses the lack of real-world data and thematic categorization for K-12 GenAI usage, providing practical insights for researchers, teachers, and students, though it is incremental in its methodological adaptation.
The authors tackled the problem of categorizing K-12 GenAI usage by analyzing over 17,000 messages from students, teachers, and ChatGPT, using a novel topic modeling approach to categorize content and tasks, revealing novel applications and insights.
We analyze anonymous interaction data of minors in class-rooms spanning several months, schools, and subjects employing a novel, simple topic modeling approach. Specifically, we categorize more than 17,000 messages generated by students, teachers, and ChatGPT in two dimensions: content (such as nature and people) and tasks (such as writing and explaining). Our hierarchical categorization done separately for each dimension includes exemplary prompts, and provides both a high-level overview as well as tangible insights. Prior works mostly lack a content or thematic categorization. While task categorizations are more prevalent in education, most have not been supported by real-world data for K-12. In turn, it is not surprising that our analysis yielded a number of novel applications. In deriving these insights, we found that many of the well-established classical and emerging computational methods, i.e., topic modeling, for analysis of large amounts of texts underperform, leading us to directly apply state-of-the-art LLMs with adequate pre-processing to achieve hierarchical topic structures with better human alignment through explicit instructions than prior approaches. Our findings support fellow researchers, teachers and students in enriching the usage of GenAI, while our discussion also highlights a number of concerns and open questions for future research.