Automatic Expert Discovery in LLM Upcycling via Sparse Interpolated Mixture-of-Experts
This addresses the need for efficient and specialized LLM adaptation, though it appears incremental as it builds on existing MoE and instruction-tuning methods.
The paper tackles the problem of fine-tuning large language models into specialized mixture-of-experts models by introducing SIMoE, which automatically discovers domain-specific experts and learns a merging strategy, achieving state-of-the-art performance on instruction-tuning benchmarks with optimal compute efficiency.
We present Sparse Interpolated Mixture-of-Experts (SIMoE) instruction-tuning, an end-to-end algorithm designed to fine-tune a dense pre-trained Large Language Model (LLM) into a MoE-style model that possesses capabilities in multiple specialized domains. During instruction-tuning, SIMoE automatically identifies multiple specialized experts under a specified sparsity constraint, with each expert representing a structurally sparse subset of the seed LLM's parameters that correspond to domain-specific knowledge within the data. SIMoE simultaneously learns an input-dependent expert merging strategy via a router network, leveraging rich cross-expert knowledge for superior downstream generalization that surpasses existing baselines. Empirically, SIMoE consistently achieves state-of-the-art performance on common instruction-tuning benchmarks while maintaining an optimal performance-compute trade-off compared to all baselines.