Markovian ODE-guided scoring can assess the quality of offline reasoning traces in language models
This addresses the need for better evaluation metrics for reasoning traces in language models, which is crucial for tasks like mathematical problem solving and fact checking, though it is incremental as it builds on existing evaluation concepts.
The paper tackled the problem of evaluating reasoning traces from language models, which existing methods fail to assess in a human-centric way, and introduced MarODE, a framework that outperformed baselines by over 250% in correlation with human judgments.
Reasoning traces produced by generative language models are increasingly used for tasks ranging from mathematical problem solving to automated fact checking. However, existing evaluation methods remain largely mechanical and fail to capture human-centric notions of reasoning quality in a way that generalizes across varied and progressively degraded reasoning. We introduce MarODE, an offline evaluation framework that assigns quality scores to reasoning traces. Its effectiveness is assessed using human-centric perturbations and human judgments, which jointly evaluate the fundamental dimensions of an evaluation metric - goodness and soundness. The approach is grounded in a Markovian formulation of reasoning progression and an ordinary differential equation based characterization of trace dynamics, enabling efficient evaluation of reasoning quality. In a large-scale evaluation, MarODE outperforms existing baselines by over 250% under Somers' D correlation. Our results emphasize the value of theory-driven evaluation frameworks as reasoning traces become central to language model-based systems.