The Unintended Trade-off of AI Alignment:Balancing Hallucination Mitigation and Safety in LLMs
This addresses a critical problem for AI safety researchers and developers by highlighting an unintended side effect in LLM alignment, though it is incremental as it builds on existing work in hallucination mitigation and safety.
The paper tackles the trade-off between improving truthfulness and maintaining safety alignment in large language models, showing that increasing factual accuracy can weaken refusal behavior, and proposes a method using sparse autoencoders and subspace orthogonalization to mitigate this issue while preserving safety.
Hallucination in large language models (LLMs) has been widely studied in recent years, with progress in both detection and mitigation aimed at improving truthfulness. Yet, a critical side effect remains largely overlooked: enhancing truthfulness can negatively impact safety alignment. In this paper, we investigate this trade-off and show that increasing factual accuracy often comes at the cost of weakened refusal behavior. Our analysis reveals that this arises from overlapping components in the model that simultaneously encode hallucination and refusal information, leading alignment methods to suppress factual knowledge unintentionally. We further examine how fine-tuning on benign datasets, even when curated for safety, can degrade alignment for the same reason. To address this, we propose a method that disentangles refusal-related features from hallucination features using sparse autoencoders, and preserves refusal behavior during fine-tuning through subspace orthogonalization. This approach prevents hallucinations from increasing while maintaining safety alignment.We evaluate our method on commonsense reasoning tasks and harmful benchmarks (AdvBench and StrongReject). Results demonstrate that our approach preserves refusal behavior and task utility, mitigating the trade-off between truthfulness and safety.