Nexus: Specialization meets Adaptability for Efficiently Training Mixture of Experts
This work addresses the problem of combining efficiency, specialization, and adaptability in large language models for users needing customizable MoE models, representing an incremental improvement with novel routing.
The paper tackles the challenge of efficiently training Mixture of Experts (MoE) models by upcycling dense expert models into an MoE to improve specialization and adaptability, achieving a 2.1% relative gain for initial upcycling and an 18.8% relative gain for extending with new experts using limited data.
Efficiency, specialization, and adaptability to new data distributions are qualities that are hard to combine in current Large Language Models. The Mixture of Experts (MoE) architecture has been the focus of significant research because its inherent conditional computation enables such desirable properties. In this work, we focus on "upcycling" dense expert models into an MoE, aiming to improve specialization while also adding the ability to adapt to new tasks easily. We introduce Nexus, an enhanced MoE architecture with adaptive routing where the model learns to project expert embeddings from domain representations. This approach allows Nexus to flexibly add new experts after the initial upcycling through separately trained dense models, without requiring large-scale MoE training for unseen data domains. Our experiments show that Nexus achieves a relative gain of up to 2.1% over the baseline for initial upcycling, and a 18.8% relative gain for extending the MoE with a new expert by using limited finetuning data. This flexibility of Nexus is crucial to enable an open-source ecosystem where every user continuously assembles their own MoE-mix according to their needs.