Reasoning-Finetuning Repurposes Latent Representations in Base Models
This provides insight into the mechanisms of reasoning fine-tuning for AI researchers, though it is incremental as it builds on prior work on steering vectors.
The study tackled the problem of understanding how reasoning fine-tuning enhances model capabilities by showing that backtracking in DeepSeek-R1-Distill-Llama-8B is driven by repurposing a pre-existing direction in base Llama-3.1-8B's activations, which systematically induces backtracking when used for steering.
Backtracking, an emergent behavior elicited by reasoning fine-tuning, has been shown to be a key mechanism in reasoning models' enhanced capabilities. Prior work has succeeded in manipulating this behavior via steering vectors, but the underlying mechanism remains poorly understood. In this work, we show that the emergence of backtracking in DeepSeek-R1-Distill-Llama-8B is in part driven by a repurposed direction already present in base model activations. Specifically, we identify a direction in base Llama-3.1-8B's residual stream which systematically induces backtracking when used to steer the distilled reasoning model, and find that the effects of steering with this direction cannot be trivially explained by token-level attributes. We further find that this direction does not induce backtracking in the base model, suggesting that the reasoning finetuning process repurposes pre-existing representations to form new behavioral circuits. Additionally, we hypothesize that this direction is one of several which may work together to mediate backtracking. Our findings offer a compelling picture that reasoning-finetuned models repurpose pre-existing base model representations, rather than learn new capabilities from scratch.