LLM-SEM: A Sentiment-Based Student Engagement Metric Using LLMS for E-Learning Platforms
This addresses the challenge of fuzzy sentiment and limited metadata in e-learning engagement analysis, though it is incremental as it applies existing LLM methods to a specific domain.
The paper tackled the problem of measuring student engagement in e-learning platforms by introducing LLM-SEM, which uses LLMs for sentiment analysis of comments and metadata normalization, resulting in a scalable and accurate metric validated through extensive experiments.
Current methods for analyzing student engagement in e-learning platforms, including automated systems, often struggle with challenges such as handling fuzzy sentiment in text comments and relying on limited metadata. Traditional approaches, such as surveys and questionnaires, also face issues like small sample sizes and scalability. In this paper, we introduce LLM-SEM (Language Model-Based Student Engagement Metric), a novel approach that leverages video metadata and sentiment analysis of student comments to measure engagement. By utilizing recent Large Language Models (LLMs), we generate high-quality sentiment predictions to mitigate text fuzziness and normalize key features such as views and likes. Our holistic method combines comprehensive metadata with sentiment polarity scores to gauge engagement at both the course and lesson levels. Extensive experiments were conducted to evaluate various LLM models, demonstrating the effectiveness of LLM-SEM in providing a scalable and accurate measure of student engagement. We fine-tuned TXLM-RoBERTa using human-annotated sentiment datasets to enhance prediction accuracy and utilized LLama 3B, and Gemma 9B from Ollama.