AntiPaSTO: Self-Supervised Steering of Moral Reasoning
This addresses the challenge of scalable moral oversight for AI systems, offering a novel approach that is incremental in improving steering methods.
The paper tackles the problem of scalable oversight for moral reasoning in large models by introducing AntiPaSTO, a self-supervised steering method that uses minimal human input (two contrasting words in templates) and achieves a 6.9x improvement over prompting baselines on DailyDilemmas while maintaining bidirectional control.
As models grow more capable, human supervision breaks down: labels don't scale, outputs can be gamed, and training doesn't generalize. Scalable oversight requires steering methods that are internal, self-supervised, and transfer out-of-distribution; existing methods satisfy some but not all three. We introduce AntiPaSTO, which separates representations along an anti-parallel axis ($α=\pm1$ produce opposite shifts), with coherence constraints preventing collapse. Human input is minimal: two contrasting words inserted into template sentences, no preference labels. Using 800 such pairs on Gemma-3-1B, AntiPaSTO beats prompting baselines by $6.9\times$ on DailyDilemmas and maintains bidirectional control where prompting triggers refusal. Code is available at https://github.com/wassname/AntiPaSTO.