Input Domain Aware MoE: Decoupling Routing Decisions from Task Optimization in Mixture of Experts
This work addresses a bottleneck in scaling large vision-language models for researchers and practitioners, offering an incremental improvement over existing sMoE methods.
The paper tackled the problem of routing mechanisms in sparse Mixture of Experts (sMoE) struggling to capture input structure, which hindered scalability and performance. They proposed Input Domain Aware MoE, a novel routing framework using a probabilistic mixture model, and demonstrated consistent outperformance over existing sMoE approaches with higher task performance and improved expert utilization balance.
Sparse Mixture of Experts (sMoE) has become a pivotal approach for scaling large vision-language models, offering substantial capacity while maintaining computational efficiency through dynamic, sparse activation of experts. However, existing routing mechanisms, typically based on similarity scoring, struggle to effectively capture the underlying input structure. This limitation leads to a trade-off between expert specialization and balanced computation, hindering both scalability and performance. We propose Input Domain Aware MoE, a novel routing framework that leverages a probabilistic mixture model to better partition the input space. By modeling routing probabilities as a mixture of distributions, our method enables experts to develop clear specialization boundaries while achieving balanced utilization. Unlike conventional approaches, our routing mechanism is trained independently of task-specific objectives, allowing for stable optimization and decisive expert assignments. Empirical results on vision-language tasks demonstrate that our method consistently outperforms existing sMoE approaches, achieving higher task performance and improved expert utilization balance.