Weight-of-Thought Reasoning: Exploring Neural Network Weights for Enhanced LLM Reasoning
This addresses the need for better reasoning capabilities in LLMs, offering improved performance and interpretability, though it appears incremental as it builds on existing prompting strategies.
The paper tackles the problem of enhancing reasoning in large language models by introducing Weight-of-Thought reasoning, which analyzes neural network weights before inference to identify pathways, resulting in superior performance on diverse tasks compared to traditional methods.
Large language models (LLMs) have demonstrated remarkable reasoning capabilities when prompted with strategies such as Chain-of-Thought (CoT). However, these approaches focus on token-level output without considering internal weight dynamics. We introduce Weight-of-Thought (WoT) reasoning, a novel approach that examines neural network weights before inference to identify reasoning pathways. Unlike existing methods, WoT explores the weight space through graph-based message passing, multi-step reasoning processes, and attention mechanisms. Our implementation creates an interconnected graph of reasoning nodes. Experiments on diverse reasoning tasks (syllogistic, mathematical, algebraic, combinatorial, and geometric) demonstrate that WoT achieves superior performance compared to traditional methods, particularly for complex problems. This approach leads to both improved performance and greater interpretability of the reasoning process, offering a promising direction for enhancing LLM reasoning capabilities.