TRACE: Toulmin-based Reasoning Assessment through Constructive Elements for LLM CoT Evaluation
For researchers evaluating open-ended LLM outputs, TRACE provides a process-oriented metric that complements accuracy-based evaluation, but it is incremental as it applies existing theories to a known problem.
TRACE introduces a metric for evaluating LLM Chain-of-Thought reasoning by analyzing argument structure using Toulmin's theory and metacognitive frameworks, achieving a 0.74 correlation with benchmark accuracy across 26.3K samples and outperforming accuracy-only baselines as a reinforcement learning reward signal.
Evaluating open-ended outputs from large language models (LLMs) remains challenging due to the absence of ground truth. Existing metrics rely on final-answer accuracy or surface-level statistics, leaving the reasoning process itself unexamined. We introduce TRACE (Toulmin-based Reasoning Assessment through Constructive Elements), a metric that analyzes Chain-of-Thought (CoT) reasoning processes. Rather than judging outcomes, TRACE inspects how arguments are constructed by integrating Toulmin's argumentation theory with Flavell's metacognitive framework to assess reasoning structure. Experiments on 26.3K QA samples across 7 reasoning models show strong correlation with benchmark accuracy (r=0.74). Furthermore, TRACE is effective as a reinforcement learning reward signal, outperforming accuracy-only baselines. Together, these results indicate that logically sound reasoning leads to higher-quality answers. TRACE thus serves as a complementary metric for evaluating open-ended outputs. Code is available at https://github.com/hyyangkisti/trace.