CircuitProbe: Predicting Reasoning Circuits in Transformers via Stability Zone Detection
This provides a practical scaling technique for small language models by enabling efficient circuit detection, though it is incremental as it builds on existing knowledge of localized reasoning circuits.
The paper tackled the problem of efficiently identifying reasoning circuits in transformer language models, which currently requires brute-force sweeps costing 25 GPU hours per model, and resulted in CircuitProbe, a method that predicts circuit locations from activation statistics in under 5 minutes on CPU, achieving a speedup of three to four orders of magnitude and matching or being within 2 layers of the optimal circuit in all validated cases across 9 models.
Transformer language models contain localized reasoning circuits, contiguous layer blocks that improve reasoning when duplicated at inference time. Finding these circuits currently requires brute-force sweeps costing 25 GPU hours per model. We propose CircuitProbe, which predicts circuit locations from activation statistics in under 5 minutes on CPU, providing a speedup of three to four orders of magnitude. We find that reasoning circuits come in two types: stability circuits in early layers, detected through the derivative of representation change, and magnitude circuits in late layers, detected through anomaly scoring. We validate across 9 models spanning 6 architectures, including 2025 models, confirming that CircuitProbe top predictions match or are within 2 layers of the optimal circuit in all validated cases. A scaling experiment across the Qwen 2.5 family reveals that layer duplication consistently benefits models under 3B parameters but degrades performance in 7B+ models, making this a practical scaling technique for small language models. CircuitProbe requires as few as 10 calibration examples and its predictions are stable across English, Hindi, Chinese, and French.