Opening the Black Box: A Survey on the Mechanisms of Multi-Step Reasoning in Large Language Models
This addresses the fundamental gap in mechanistic understanding of LLM reasoning for researchers, though it is an incremental contribution as a survey rather than new empirical findings.
This survey tackles the problem of understanding the internal mechanisms that enable large language models to perform multi-step reasoning, providing a comprehensive overview organized around seven research questions about implicit and explicit reasoning processes.
Large Language Models (LLMs) have demonstrated remarkable abilities to solve problems requiring multiple reasoning steps, yet the internal mechanisms enabling such capabilities remain elusive. Unlike existing surveys that primarily focus on engineering methods to enhance performance, this survey provides a comprehensive overview of the mechanisms underlying LLM multi-step reasoning. We organize the survey around a conceptual framework comprising seven interconnected research questions, from how LLMs execute implicit multi-hop reasoning within hidden activations to how verbalized explicit reasoning remodels the internal computation. Finally, we highlight five research directions for future mechanistic studies.