Navigating the Alignment-Calibration Trade-off: A Pareto-Superior Frontier via Model Merging
This addresses the alignment-calibration trade-off for AI model developers, offering a computationally efficient method to mitigate the full scope of the alignment tax, though it is incremental as it builds on existing model merging techniques.
The paper tackles the problem of the alignment tax in post-training, which causes a drop in task accuracy and severe loss of calibration, making models overconfident and less reliable. It shows that interpolating between a model's weights before and after alignment yields Pareto-optimal models that improve accuracy beyond both parents and substantially recover calibration.
The "alignment tax" of post-training is typically framed as a drop in task accuracy. We show it also involves a severe loss of calibration, making models overconfident, less reliable, and model outputs less diverse. We show that this trade-off can be navigated effectively via a simple post-hoc intervention: interpolating between a model's weights before and after alignment. Crucially, this is not a strict trade-off. We find that the process consistently reveals Pareto-optimal interpolations - models that improve accuracy beyond both parents while substantially recovering the calibration lost during alignment. Our work demonstrates that simple model merging provides a computationally efficient method for mitigating the full scope of the alignment tax, yielding models that are more capable and more reliable.