Ruffle&Riley: Towards the Automated Induction of Conversational Tutoring Systems
This work addresses the scalability issue for educational technology developers by automating CTS creation, though it shows incremental improvements in user experience without significant learning gains.
The paper tackled the problem of high authoring costs for conversational tutoring systems by introducing a system that automatically induces tutoring scripts from lesson text and orchestrates them using LLM-based agents in a learning-by-teaching format. In a user study with 100 participants, it found no significant differences in post-test scores compared to simpler methods, but users reported higher ratings for understanding, remembering, helpfulness, and coherence.
Conversational tutoring systems (CTSs) offer learning experiences driven by natural language interaction. They are known to promote high levels of cognitive engagement and benefit learning outcomes, particularly in reasoning tasks. Nonetheless, the time and cost required to author CTS content is a major obstacle to widespread adoption. In this paper, we introduce a novel type of CTS that leverages the recent advances in large language models (LLMs) in two ways: First, the system induces a tutoring script automatically from a lesson text. Second, the system automates the script orchestration via two LLM-based agents (Ruffle&Riley) with the roles of a student and a professor in a learning-by-teaching format. The system allows a free-form conversation that follows the ITS-typical inner and outer loop structure. In an initial between-subject online user study (N = 100) comparing Ruffle&Riley to simpler QA chatbots and reading activity, we found no significant differences in post-test scores. Nonetheless, in the learning experience survey, Ruffle&Riley users expressed higher ratings of understanding and remembering and further perceived the offered support as more helpful and the conversation as coherent. Our study provides insights for a new generation of scalable CTS technologies.