MathBuddy: A Multimodal System for Affective Math Tutoring
This addresses the need for more effective and empathetic AI tutors in education, though it is incremental as it builds on existing LLM and multimodal techniques.
The authors tackled the problem of LLM-based educational systems ignoring student emotions by developing MathBuddy, a multimodal math tutor that models emotions from text and facial expressions to provide empathetic responses, resulting in a 23-point win rate gain and a 3-point DAMR score improvement.
The rapid adoption of LLM-based conversational systems is already transforming the landscape of educational technology. However, the current state-of-the-art learning models do not take into account the student's affective states. Multiple studies in educational psychology support the claim that positive or negative emotional states can impact a student's learning capabilities. To bridge this gap, we present MathBuddy, an emotionally aware LLM-powered Math Tutor, which dynamically models the student's emotions and maps them to relevant pedagogical strategies, making the tutor-student conversation a more empathetic one. The student's emotions are captured from the conversational text as well as from their facial expressions. The student's emotions are aggregated from both modalities to confidently prompt our LLM Tutor for an emotionally-aware response. We have evaluated our model using automatic evaluation metrics across eight pedagogical dimensions and user studies. We report a massive 23 point performance gain using the win rate and a 3 point gain at an overall level using DAMR scores which strongly supports our hypothesis of improving LLM-based tutor's pedagogical abilities by modeling students' emotions. Our dataset and code are available at: https://github.com/ITU-NLP/MathBuddy .