Post-Hoc Merging is Not Enough: Many-Shot Model Merging with Loss-Gap Balancing
For practitioners combining multiple task-specialized LLMs, this provides a more effective merging strategy that reduces information erasure.
This work shows that post-hoc one-shot model merging suffers from task interference and proposes METIS, an iterative many-shot merging method with loss-gap weighting and consensus masking, which significantly improves multi-task LLM performance, especially on the worst-performing task.
Model merging has become a practical post-training strategy for building a single multi-task large language model (LLM) by combining multiple task-specialized models. However, most existing approaches rely on post-hoc merging, in which task-specific models are merged only once after training. This one-shot aggregation often suffers from task interference, leading to information erasure across individual tasks. In this work, we show that replacing post-hoc merging with an iterative many-shot merging protocol is effective in improving multi-task performance. Building on this insight, we propose METIS, Mitigating Erasure from Task Interference for Stable many-shot merging. METIS is a loss-aware many-shot merging method that addresses information erasure in post-hoc merging through task-wise loss-gap weighting and consensus-based masking. Notably, METIS exhibits significant performance improvement on the worst-performing task, effectively mitigating information erasure. (Project page: https://imkyungjin.github.io/METIS/)