MoD: A Distribution-Based Approach for Merging Large Language Models
This work addresses resource and operational efficiency problems for developers and users of multiple specialized LLMs, though it appears incremental as it builds on existing model merging techniques.
The paper tackles the challenge of efficiently merging specialized large language models (LLMs) by proposing the Mixture of Distributions (MoD) framework, which operates on output probability distributions instead of model weights, and demonstrates that it significantly outperforms existing merging techniques on mathematical reasoning benchmarks using Qwen2.5 models.
Large language models (LLMs) have enabled the development of numerous specialized, task-specific variants. However, the maintenance and deployment of these individual models present substantial challenges in terms of resource utilization and operational efficiency. In this work, we propose the \textit{Mixture of Distributions (MoD)} framework, a novel approach for merging LLMs that operates directly on their output probability distributions, rather than on model weights. Unlike traditional weight-averaging methods, MoD effectively preserves the specialized capabilities of individual models while enabling efficient knowledge sharing across tasks. Through extensive experimentation on mathematical reasoning benchmarks using Qwen2.5 models, we demonstrate that MoD significantly outperforms existing model merging techniques across multiple benchmarks. All code, data, and experimental materials are published at https://github.com/knovel-eng/mod.