Trade-offs in Ensembling, Merging and Routing Among Parameter-Efficient Experts
This work addresses the challenge of efficiently combining models for multi-task learning, which is incremental as it builds on existing ensembling, merging, and routing strategies.
The paper tackled the problem of fusing independently trained lightweight adapters for multi-task learning in large language models, finding that non-uniform ensembling and merging improve performance, but routing offers even greater gains, with techniques like clustering and greedy subset selection reducing computational overhead.
While large language models (LLMs) fine-tuned with lightweight adapters achieve strong performance across diverse tasks, their performance on individual tasks depends on the fine-tuning strategy. Fusing independently trained models with different strengths has shown promise for multi-task learning through three main strategies: ensembling, which combines outputs from independent models; merging, which fuses model weights via parameter averaging; and routing, which integrates models in an input-dependent fashion. However, many design decisions in these approaches remain understudied, and the relative benefits of more sophisticated ensembling, merging and routing techniques are not fully understood. We empirically evaluate their trade-offs, addressing two key questions: What are the advantages of going beyond uniform ensembling or merging? And does the flexibility of routing justify its complexity? Our findings indicate that non-uniform ensembling and merging improve performance, but routing offers even greater gains. To mitigate the computational cost of routing, we analyze expert selection techniques, showing that clustering and greedy subset selection can maintain reasonable performance with minimal overhead. These insights advance our understanding of model fusion for multi-task learning.