H3Fusion: Helpful, Harmless, Honest Fusion of Aligned LLMs
This work addresses the problem of ensuring safe and effective LLM responses for users, representing an incremental improvement over existing alignment methods.
The paper tackles the challenge of aligning pre-trained LLMs to be helpful, harmless, and honest simultaneously by proposing H3Fusion, a mixture-of-experts-based fusion mechanism that models alignment as a controllable drift, resulting in outperforming individually aligned models by 11.37% and improving robustness over state-of-the-art ensemble and model-merging approaches by 13.77% and 6.18%, respectively.
The alignment of pre-trained LLMs continues to draw significant attention from both industry and academia, aiming to ensure responses that are helpful, harmless, and honest. However, identifying a point in the model's representation subspace that simultaneously satisfies all these properties remains challenging. H3Fusion addresses this challenge by introducing a mixture-of-experts (MoE)-based fusion mechanism that models alignment as a controllable drift within the subspace, guided by a drift-regularization loss to balance competing alignment dimensions. Furthermore, we formulate the alignment by finding a dual objective of harnessing the distance of generated embeddings and alignment embeddings, and introduce a gating loss by canalizing the activations on the contributing experts. Extensive evaluations of three benchmark datasets show that H3Fusion is more helpful, less harmful, and more honest in three aspects: it outperforms each individually aligned model by 11.37%, and provides stronger robustness compared to the state-of-the-art LLM ensemble approaches by 13.77% and model-merging approaches by 6.18%. Code is available at https://github.com/sftekin/h3fusion.