ThinkProbe: Beyond Accuracy -- Structural Profiling of Open-Ended LLM Reasoning Traces via Non-Generative Thought Graphs
For researchers evaluating LLM reasoning, ThinkProbe provides a structural profiling method that exposes cognitive differences invisible to accuracy-based metrics, though it is an incremental tool for analysis rather than a new paradigm.
ThinkProbe introduces a non-generative framework to convert LLM reasoning traces into Thought Graphs with a 19-metric cognitive profile, revealing that reasoning structure is a stable model-level property with between-model variance exceeding between-domain variance up to fourfold across four of five dimensions.
We present ThinkProbe, a framework for structural analysis of LLM reasoning traces. ThinkProbe converts each trace into a Thought Graph a directed graph with cycles, 8 node types, and 6 edge types and derives a 19-metric five-dimensional cognitive profile (5D-CP: Breadth, Depth, Structure, Metacognitive, Efficiency) through a fully non-generative pipeline combining rule-based segmentation and discriminative semantic linking. Applied to 4{,}200 traces from 7 native reasoning models across 200 open-ended questions and 10 cognitive domains, ThinkProbe reveals that reasoning structure is a stable, model-level property: between-model variance exceeds between-domain variance by up to fourfold across four of five cognitive dimensions, with Structure showing genuine sensitivity to question domain, exposing qualitatively distinct cognitive profiles invisible to accuracy-based evaluation.