Functionality-Oriented LLM Merging on the Fisher--Rao Manifold
This work addresses the problem of effectively combining multiple LLMs for practitioners and researchers, offering a more robust and principled approach to model merging.
This paper tackles the problem of merging multiple fine-tuned LLMs into a single model without retraining, which is often limited by Euclidean parameter-space heuristics. By formulating model merging as computing a weighted Karcher mean on the Fisher--Rao manifold, the authors developed a fixed-point algorithm that consistently outperforms prior baselines across various benchmarks and collapse diagnostics, remaining stable even with an increased number and heterogeneity of merged models.
Weight-space merging aims to combine multiple fine-tuned LLMs into a single model without retraining, yet most existing approaches remain fundamentally parameter-space heuristics. This creates three practical limitations. First, linear averaging, task vectors, and related rules operate on Euclidean coordinates, even though the desired goal is to merge functionality, i.e., predictive behaviors across tasks. Second, when the source checkpoints are farther apart or more heterogeneous, Euclidean blends often trigger representation collapse, manifested as activation variance shrinkage and effective-rank degradation, which sharply degrades accuracy. Third, many geometry-inspired methods are most natural for two-model interpolation and do not extend cleanly to merging N>2 experts with a principled objective. We address these issues by formulating model merging as computing a weighted Karcher mean on the Fisher--Rao manifold, which is locally equivalent to minimizing a KL-based function distance between predictive distributions. We derive a practical fixed-point algorithm using a lightweight spherical proxy that preserves norms and generalizes directly to multi-expert merging. Across various benchmarks and collapse diagnostics, our method remains stable as the number and heterogeneity of merged models increase, consistently outperforming prior baselines.