A Modular Dataset to Demonstrate LLM Abstraction Capability
This work addresses the problem of improving AI reliability and interpretability for researchers and developers by providing insights into LLM reasoning mechanisms, though it is incremental as it builds on existing methods for analyzing internal representations.
The authors tackled the problem of understanding LLM reasoning errors by introducing the ArrangementPuzzle dataset and found that a classifier on LLM activations achieved over 80% accuracy in predicting reasoning correctness, indicating internal distinctions between correct and incorrect steps.
Large language models (LLMs) exhibit impressive capabilities but struggle with reasoning errors due to hallucinations and flawed logic. To investigate their internal representations of reasoning, we introduce ArrangementPuzzle, a novel puzzle dataset with structured solutions and automated stepwise correctness verification. We trained a classifier model on LLM activations on this dataset and found that it achieved over 80% accuracy in predicting reasoning correctness, implying that LLMs internally distinguish between correct and incorrect reasoning steps, with the strongest representations in middle-late Transformer layers. Further analysis reveals that LLMs encode abstract reasoning concepts within the middle activation layers of the transformer architecture, distinguishing logical from semantic equivalence. These findings provide insights into LLM reasoning mechanisms and contribute to improving AI reliability and interpretability, thereby offering the possibility to manipulate and refine LLM reasoning.