Tell Me Why: Designing an Explainable LLM-based Dialogue System for Student Problem Behavior Diagnosis
For educators using LLM-based diagnostic tools, this work addresses the need for transparency and trust by providing explanations for recommendations.
The paper presents an explainable dialogue system for diagnosing student problem behaviors, using a hierarchical attribution method to generate natural-language explanations for LLM recommendations. In a user study with 22 pre-service teachers, explanations increased trust in the system.
Diagnosing student problem behaviors requires teachers to synthesize multifaceted information, identify behavioral categories, and plan intervention strategies. Although fine-tuned large language models (LLMs) can support this process through multi-turn dialogue, they rarely explain why a strategy is recommended, limiting transparency and teachers' trust. To address this issue, we present an explainable dialogue system built on a fine-tuned LLM. The system uses a hierarchical attribution method based on explainable AI (xAI) to identify dialogue evidence for each recommendation and generate a natural-language explanation based on that evidence. In technical evaluation, the method outperformed baseline approaches in identifying supporting evidence. In a preliminary user study with 22 pre-service teachers, participants who received explanations reported higher trust in the system. These findings suggest a promising direction for improving LLM explainability in educational dialogue systems.