Long-Tailed Distribution-Aware Router For Mixture-of-Experts in Large Vision-Language Model
This work addresses routing inefficiencies in large vision-language models, offering a domain-specific improvement for multimodal AI applications.
The paper tackled the problem of token-to-expert routing in mixture-of-experts for large vision-language models by proposing a Long-Tailed Distribution-aware Router (LTDR), which addresses distributional differences between vision and language modalities and enhances expert activation for vision tail tokens, achieving improved performance on benchmarks.
The mixture-of-experts (MoE), which replaces dense models with sparse architectures, has gained attention in large vision-language models (LVLMs) for achieving comparable performance with fewer activated parameters. Existing MoE frameworks for LVLMs focus on token-to-expert routing (TER), encouraging different experts to specialize in processing distinct tokens. However, these frameworks often rely on the load balancing mechanism, overlooking the inherent distributional differences between vision and language. To this end, we propose a Long-Tailed Distribution-aware Router (LTDR) for vision-language TER, tackling two challenges: (1) Distribution-aware router for modality-specific routing. We observe that language TER follows a uniform distribution, whereas vision TER exhibits a long-tailed distribution. This discrepancy necessitates distinct routing strategies tailored to each modality. (2) Enhancing expert activation for vision tail tokens. Recognizing the importance of vision tail tokens, we introduce an oversampling-like strategy by increasing the number of activated experts for these tokens. Experiments on extensive benchmarks validate the effectiveness of our approach.