Paying Alignment Tax with Contrastive Learning
This addresses the alignment tax problem in language model debiasing, offering a solution to reduce bias while preserving model capabilities, which is incremental as it builds on existing debiasing methods but introduces a novel framework to overcome their limitations.
The paper tackles the problem of debiasing language models without degrading their capabilities, such as factual accuracy and knowledge retention, by proposing a contrastive learning framework that achieves substantial improvements in both toxicity reduction and faithfulness preservation, consistently improving both metrics simultaneously across multiple model scales.
Current debiasing approaches often result a degradation in model capabilities such as factual accuracy and knowledge retention. Through systematic evaluation across multiple benchmarks, we demonstrate that existing debiasing methods face fundamental trade-offs, particularly in smaller models, leading to reduced truthfulness, knowledge loss, or unintelligible outputs. To address these limitations, we propose a contrastive learning framework that learns through carefully constructed positive and negative examples. Our approach introduces contrast computation and dynamic loss scaling to balance bias mitigation with faithfulness preservation. Experimental results across multiple model scales demonstrate that our method achieves substantial improvements in both toxicity reduction and faithfulness preservation. Most importantly, we show that our framework is the first to consistently improve both metrics simultaneously, avoiding the capability degradation characteristic of existing approaches. These results suggest that explicit modeling of both positive and negative examples through contrastive learning could be a promising direction for reducing the alignment tax in language model debiasing.