Leveraging Large Language Models for Identifying Knowledge Components
This addresses the bottleneck of manual KC identification for adaptive learning systems, but is incremental as it builds on prior LLM approaches with a refinement technique.
The study tackled the problem of automating knowledge component identification for adaptive learning systems by scaling an LLM prompting approach to 646 multiple-choice questions, finding it performed worse than experts (RMSE 0.4285 vs. 0.4206) and generated too many KCs (569 vs. 101), but a novel semantic merging method improved performance to RMSE 0.4259 with 428 KCs.
Knowledge Components (KCs) are foundational to adaptive learning systems, but their manual identification by domain experts is a significant bottleneck. While Large Language Models (LLMs) offer a promising avenue for automating this process, prior research has been limited to small datasets and has been shown to produce superfluous, redundant KC labels. This study addresses these limitations by first scaling a "simulated textbook" LLM prompting strategy (using GPT-4o-mini) to a larger dataset of 646 multiple-choice questions. We found that this initial automated approach performed significantly worse than an expert-designed KC model (RMSE 0.4285 vs. 0.4206) and generated an excessive number of KCs (569 vs. 101). To address the issue of redundancy, we proposed and evaluated a novel method for merging semantically similar KC labels based on their cosine similarity. This merging strategy significantly improved the model's performance; a model using a cosine similarity threshold of 0.8 achieved the best result, reducing the KC count to 428 and improving the RMSE to 0.4259. This demonstrates that while scaled LLM generation alone is insufficient, combining it with a semantic merging technique offers a viable path toward automating and refining KC identification.