Towards Reversible Model Merging For Low-rank Weights
This addresses the challenge of model merging for low-rank weights in machine learning, which is incremental as it builds on prior merging work but introduces a novel reversible approach.
The paper tackles the problem of merging multiple fine-tuned models into a single set of weights, particularly when models are compressed into low-rank representations, by proposing Reversible Model Merging (RMM) to construct a compact basis that allows recovery of original task-specific models via linear combination, resulting in consistent outperformance of existing merging approaches and significant preservation of performance for low-rank compressed models.
Model merging aims to combine multiple fine-tuned models into a single set of weights that performs well across all source tasks. While prior work has shown that merging can approximate the performance of individual fine-tuned models for each task, it largely overlooks scenarios where models are compressed into low-rank representations, either through low-rank adaptation (LoRA) or post-training singular value decomposition (SVD). We first demonstrate that applying conventional merging methods to low-rank weights leads to severe performance degradation in the merged model. Motivated by this phenomenon, we propose a fundamentally different approach: instead of collapsing all adapters into one set of weights, we construct a compact basis (e.g., an equivalent of holding two or more models) from which original task-specific models can be recovered via linear combination. This reframes merging as generating a reconstruction-capable model space rather than producing a single merged model. Crucially, this allows us to ``revert'' to each individual model when needed, recognizing that no merged model can consistently outperform one specialized for its task. Building on this insight, we introduce our method, Reversible Model Merging (RMM), an efficient, data-free, and flexible method that provides a closed-form solution for selecting the optimal basis of model weights and task-specific coefficients for linear combination. Extensive experiments across diverse datasets and model scales demonstrate that RMM consistently outperforms existing merging approaches, preserving the performance of low-rank compressed models by a significant margin.