Lynx: Enabling Efficient MoE Inference through Dynamic Batch-Aware Expert Selection
This addresses a critical bottleneck for deploying efficient large language models in production, representing an incremental improvement over existing MoE methods.
The paper tackles the inefficiency of Mixture-of-Experts (MoE) models during batched inference by introducing Lynx, a system that dynamically selects experts based on batch context, achieving up to 1.55x reduction in inference latency with negligible accuracy loss.
Mixture-of-Experts (MoE) architectures have recently gained popularity in enabling efficient scaling of large language models. However, we uncover a fundamental tension: while MoEs are designed for selective expert activation, production serving requires request batching, which forces the activation of all experts and negates MoE's efficiency benefits during the decode phase. We present Lynx, a system that enables efficient MoE inference through dynamic, batch-aware expert selection. Our key insight is that expert importance varies significantly across tokens and inference phases, creating opportunities for runtime optimization. Lynx leverages this insight through a lightweight framework that dynamically reduces active experts while preserving model accuracy. Our evaluations show that Lynx achieves up to 1.55x reduction in inference latency while maintaining negligible accuracy loss from baseline model across complex code generation and mathematical reasoning tasks.