LLM Safety Alignment is Divergence Estimation in Disguise
This provides a theoretical foundation for LLM safety alignment, potentially improving methods for AI safety researchers, though it appears incremental as it builds on existing alignment approaches.
The authors demonstrate that LLM safety alignment methods like RLHF can be framed as divergence estimation between aligned and unaligned distributions, explaining latent space separation. They propose KLDO, a KL divergence-based method, and show that compliance-refusal datasets improve safety alignment, with a new distance metric quantifying separation.
We present a theoretical framework showing that popular LLM alignment methods, including RLHF and its variants, can be understood as divergence estimators between aligned (safe or preferred) and unaligned (harmful or less preferred) distributions. This perspective explains the emergence of separation in the latent space between safe and harmful prompts after alignment. As an application of our general divergence framework, we propose KLDO, a novel KL divergence-based alignment method, and empirically validate its effectiveness. We further show that using compliance-refusal datasets, rather than standard preference-based datasets, leads to stronger separation and improved safety alignment. Finally, to quantify the separation effect, we propose a distance-based metric in the prompt representation space, which also acts as a statistically significant indicator for model safety.