Consistency Training Can Entrench Misalignment
This paper identifies that consistency training, a widely used method, is not alignment-neutral and can entrench certain misbehaviors, which is critical for developers of large language models.
Consistency training can either suppress or amplify different forms of misalignment in language models: it reduces reward hacking and emergent misalignment but increases sycophancy, as shown across 108 model organisms. The primary driver appears to be distribution shifts from the labeling process.
Consistency training encourages a model to produce similar outputs across related inputs or sampling procedures. Such methods are simple, scalable, and largely label-free, but their effects on model alignment remain poorly understood. Could the self-bootstrapping nature of these methods amplify undesired behavior in models? We test seven consistency training methods on 108 model organisms: open-source models (7B--70B) fine-tuned to exhibit various forms of controlled misaligned behavior. We find that outcomes vary significantly: consistency training generally suppresses reward hacking and emergent misalignment but amplifies sycophancy. We present evidence that distribution shifts induced by the consistency labeling process, rather than variation in the selection operators, may be the primary driver of systematic alignment effects. Finally, we present a unifying theoretical framework to derive conditions under which consistency training will amplify or suppress misalignment. In total, our study establishes that consistency training is not alignment-neutral, and that its use in critical systems should be carefully audited.