Preference Tuning as Spectral Update Reorganization
For researchers in alignment and preference optimization, this provides a mechanistic understanding of how updates are organized, suggesting that alignment gain and coverage loss are tied to spectral structure, which is an incremental but insightful contribution.
The paper reveals that preference optimization (e.g., RLHF) induces a spectral head-tail organization in parameter updates, where the head dominates endpoint behavior but the tail is necessary for full solution recovery, especially on out-of-distribution tasks. This recasts alignment as structured update reorganization rather than monolithic correction.
Preference-based post-training is usually understood through endpoint behavior, yet the learned update that produces this behavior remains largely opaque. We study RLHF and related preference optimization through the spectral structure of their induced parameter updates. By decomposing effective LoRA updates and reloading their spectral components as plug-in modules, we turn preference-induced updates into objects that can be isolated, recomposed, and directly intervened on. Across model families, optimization algorithms, and supervision regimes, these updates consistently develop a spectral head--tail organization. A compact head emerges early and carries the dominant endpoint shift, while a heterogeneous residual tail remains. The split is functional rather than merely descriptive. Plug-in intervention shows that the head accounts for the visible behavioral departure from the base model, while the tail is weak in isolation. Cross-run recomposition further shows that mixed adapters follow the source of the head, indicating that the head carries run-level solver bias. This endpoint dominance does not imply learning sufficiency. Head-only learning is non-vacuous but fails to recover the full solution, especially on out-of-distribution behavior. Tail-only learning yields little visible gain, yet the full solution is not recovered without the tail. These findings recast preference post-training as structured update reorganization rather than a monolithic behavioral correction, and suggest that alignment gain and coverage loss are tied to how the learned update itself is organized.