Weight Updates as Activation Shifts: A Principled Framework for Steering
This work addresses the need for parameter-efficient adaptation in machine learning, offering a novel paradigm that combines weight and activation updates to surpass performance ceilings of existing methods.
The paper tackled the problem of designing effective activation steering for efficient model adaptation by establishing a first-order equivalence between activation-space interventions and weight-space updates, which led to a principled framework and a new joint adaptation approach. The result was post-block steering achieving accuracy within 0.2%-0.9% of full-parameter tuning on average across tasks and models while training only 0.04% of parameters, outperforming prior methods like ReFT and LoRA.
Activation steering promises to be an extremely parameter-efficient form of adaptation, but its effectiveness depends on critical design choices -- such as intervention location and parameterization -- that currently rely on empirical heuristics rather than a principled foundation. We establish a first-order equivalence between activation-space interventions and weight-space updates, deriving the conditions under which activation steering can replicate fine-tuning behavior. This equivalence yields a principled framework for steering design and identifies the post-block output as a theoretically-backed and highly expressive intervention site. We further explain why certain intervention locations outperform others and show that weight updates and activation updates play distinct, complementary functional roles. This analysis motivates a new approach -- joint adaptation -- that trains in both spaces simultaneously. Our post-block steering achieves accuracy within 0.2%-0.9%$ of full-parameter tuning, on average across tasks and models, while training only 0.04% of model parameters. It consistently outperforms prior activation steering methods such as ReFT and PEFT approaches including LoRA, while using significantly fewer parameters. Finally, we show that joint adaptation often surpasses the performance ceilings of weight and activation updates in isolation, introducing a new paradigm for efficient model adaptation.