Measuring Reasoning in LLMs: a New Dialectical Angle
This work addresses the need for more rigorous evaluation of reasoning processes in LLMs, offering a novel approach that could impact AI benchmarking, though it is incremental in focusing on assessment rather than model improvement.
The paper tackled the problem of evaluating reasoning in language models by proposing a dialectical framework called SIEV, which assesses the dynamic process of reasoning rather than just correctness, revealing that state-of-the-art models like GPT-5-chat lose over 40 points on GSM when evaluated this way.
What does it truly mean for a language model to "reason"? Most current evaluations and benchmarks reward models' correct standalone answers--but correctness alone reveals little about the process that produced them. In this work, we explore a different perspective: reasoning is not a static chain of steps, but a dynamic trajectory where ideas interact, clash, and evolve into deeper insights. To capture this dynamic, we draw on a well-established philosophical tradition: \textit{dialectics}, where reasoning unfolds through thesis, antithesis, and synthesis. Building on this, we present SIEV, a structured framework that evaluates reasoning of LLMs through dialectics. Unlike conventional evaluations, SIEV assesses not only the conclusion a model reaches, but how it gets there: its ability to resolve tension, integrate distinct ideas, and synthesize higher-order reasoning. This lens uncovers significant reasoning gaps in state-of-the-art models even under saturated benchmarks like GSM and MMLU. For instance, GPT-5-chat, a recent model, loses over 40 points (out of 100) when evaluated with SIEV on GSM. Our findings highlight that adopting a process-oriented, philosophically grounded approach enables a deeper, more rigorous, and more discriminative assessment of LLM reasoning.