Alignment-Aware Model Adaptation via Feedback-Guided Optimization
This addresses the issue of alignment degradation during fine-tuning for users of foundation models, offering an incremental improvement over standard methods.
The paper tackles the problem of fine-tuning foundation models for downstream tasks while preserving alignment objectives like safety and hallucination avoidance, proposing a framework that integrates external alignment feedback via policy-gradient regularization and adaptive gating, resulting in consistent reductions in harmful and hallucinated outputs without sacrificing task performance.
Fine-tuning is the primary mechanism for adapting foundation models to downstream tasks; however, standard approaches largely optimize task objectives in isolation and do not account for secondary yet critical alignment objectives (e.g., safety and hallucination avoidance). As a result, downstream fine-tuning can degrade alignment and fail to correct pre-existing misaligned behavior. We propose an alignment-aware fine-tuning framework that integrates feedback from an external alignment signal through policy-gradient-based regularization. Our method introduces an adaptive gating mechanism that dynamically balances supervised and alignment-driven gradients on a per-sample basis, prioritizing uncertain or misaligned cases while allowing well-aligned examples to follow standard supervised updates. The framework further learns abstention behavior for fully misaligned inputs, incorporating conservative responses directly into the fine-tuned model. Experiments on general and domain-specific instruction-tuning benchmarks demonstrate consistent reductions in harmful and hallucinated outputs without sacrificing downstream task performance. Additional analyses show robustness to adversarial fine-tuning, prompt-based attacks, and unsafe initializations, establishing adaptively gated alignment optimization as an effective approach for alignment-preserving and alignment-recovering model adaptation.