A Coin Flip Per Token: Bernoulli Sparse Steering of Large Language Models
For practitioners of LLM behavioral control, this work demonstrates that dense token-level intervention is unnecessary, reducing computational cost and improving fluency.
The authors show that steering only 50% of tokens with sparse autoencoder-based activation steering recovers most of the effect of dense steering while preserving fluency, and that steering as few as 30% of tokens outperforms prompt-based control.
Activation steering via sparse autoencoders (SAEs) enables behavioral control of large language models without task-specific fine-tuning, but standard methods apply the steering signal at every generated token, incurring constant per-token perturbation that risks degrading fluency. We ask: is dense intervention necessary? We introduce Stochastic Token Steering (STS), which gates each token independently with probability $p$, and Stochastic Block Steering (SBS), which gates a leading window once per sequence; neither requires a reward model or learned gating policy. Across two model families and two behavioral tasks, steering only 50% of the tokens recovers most of the dense-steering effect while preserving fluency, and steering as few as 30% surpasses prompt-based control. The optimal steering magnitude scales inversely with the intervention ratio, revealing that SAE-mediated control is rate-limited: the behavioral outcome depends on cumulative signal dosage across a sequence.