Markos Stamatakis

Markos Stamatakis, Omkar Gavali, Joshua Berger et al.

Chatbots based on large language models (LLMs) are increasingly adopted for information retrieval, text generation, and writing assistance. In educational settings, their use is also rapidly increasing. Students leverage these systems to complete tasks, access information, and support learning. However, the role of LLM-based chatbots in supporting learning and assessment in university-level computer science education is still underexplored. To address this gap, we investigate the performance of several LLM-based chatbots in solving multiple-choice questions (MCQs) at the university level and evaluate their capabilities to assist student learning. We developed 70 MCQs for a university lecture on interactive visual data analysis and evaluated the chatbots' performance using different prompt designs. We further compared the results with students' performance. Finally, we conducted a user study in two lectures (interactive visual data analysis, computer vision) to investigate how chatbot-generated answers and explanations affect students' performance. The chatbot performance showed significant differences between smaller models and GPT-4o and GPT-5 models, which achieved the best results. The results of the user study show that presenting ChatGPT answers together with an explanation does not improve students' performance in general.

Markos Stamatakis, Joshua Berger, Christian Wartena et al.

Web-based educational videos offer flexible learning opportunities and are becoming increasingly popular. However, improving user engagement and knowledge retention remains a challenge. Automatically generated questions can activate learners and support their knowledge acquisition. Further, they can help teachers and learners assess their understanding. While large language and vision-language models have been employed in various tasks, their application to question generation for educational videos remains underexplored. In this paper, we investigate the capabilities of current vision-language models for generating learning-oriented questions for educational video content. We assess (1) out-of-the-box models' performance; (2) fine-tuning effects on content-specific question generation; (3) the impact of different video modalities on question quality; and (4) in a qualitative study, question relevance, answerability, and difficulty levels of generated questions. Our findings delineate the capabilities of current vision-language models, highlighting the need for fine-tuning and addressing challenges in question diversity and relevance. We identify requirements for future multimodal datasets and outline promising research directions.