FeatCal: Feature Calibration for Post-Merging Models
For practitioners merging task-specific models, FeatCal offers a fast, sample-efficient calibration method that significantly reduces the performance gap between merged and expert models.
FeatCal calibrates merged model weights layer by layer to reduce feature drift, achieving 85.5% accuracy on CLIP-ViT-B/32 Task Arithmetic vs. 77.0% for Surgery and 78.8% for ProbSurgery, and 85.2% on FLAN-T5-base GLUE vs. 83.7% and 82.2%, with 4x faster calibration.
Model merging combines task experts into one model and avoids joint training, retraining, or deploying many expert models, but the merged model often still underperforms task experts. We study this performance gap through feature drift, the difference between features produced by the merged model and by the expert on the same input. Our theory decomposes this drift into upstream propagation and local mismatch, tracks how it propagates and combines through later layers in forward order, and links final feature drift to output drift. This view motivates FeatCal, which uses a small calibration set to calibrate the merged model weights layer by layer in forward order, reducing feature drift while staying close to merged weights and preserving the benefits of model merging. FeatCal uses an efficient closed-form solution to update model weights, with no gradient descent, iterative optimization, or extra modules. On the main CLIP and GLUE benchmarks, FeatCal beats Surgery and ProbSurgery, the closest post-merging calibration baselines: 85.5% vs. 77.0%/78.8% on CLIP-ViT-B/32 Task Arithmetic (TA) and 85.2% vs. 83.7%/82.2% on FLAN-T5-base GLUE. On CLIP-ViT-B/32, 8 examples per task reach 82.9%, and 256 examples per task take 53 seconds, about 4x faster than both baselines, showing better sample efficiency and lower calibration cost.