When the Chain Breaks: Interactive Diagnosis of LLM Chain-of-Thought Reasoning Errors
This addresses the challenge of trust calibration for users of LLMs by improving interpretability of reasoning traces, though it is incremental as it builds on existing error detection methods.
The paper tackles the problem of interpreting lengthy and error-prone Chain-of-Thought reasoning traces from Large Language Models by developing ReasonDiag, an interactive visualization system that helps users understand traces, identify errors, and determine root causes, as shown in user interviews with 16 participants.
Current Large Language Models (LLMs), especially Large Reasoning Models, can generate Chain-of-Thought (CoT) reasoning traces to illustrate how they produce final outputs, thereby facilitating trust calibration for users. However, these CoT reasoning traces are usually lengthy and tedious, and can contain various issues, such as logical and factual errors, which make it difficult for users to interpret the reasoning traces efficiently and accurately. To address these challenges, we develop an error detection pipeline that combines external fact-checking with symbolic formal logical validation to identify errors at the step level. Building on this pipeline, we propose ReasonDiag, an interactive visualization system for diagnosing CoT reasoning traces. ReasonDiag provides 1) an integrated arc diagram to show reasoning-step distributions and error-propagation patterns, and 2) a hierarchical node-link diagram to visualize high-level reasoning flows and premise dependencies. We evaluate ReasonDiag through a technical evaluation for the error detection pipeline, two case studies, and user interviews with 16 participants. The results indicate that ReasonDiag helps users effectively understand CoT reasoning traces, identify erroneous steps, and determine their root causes.