Stable Routing for Mixture-of-Experts in Class-Incremental Learning
For researchers working on class-incremental learning with pre-trained models, this paper identifies and solves a specific bottleneck (routing drift) in expandable MoE, offering a practical improvement over existing methods.
The paper addresses routing drift in mixture-of-experts (MoE) for class-incremental learning (CIL), where expert expansion causes the router to reassign old-class samples to new experts, degrading performance. They propose StaR-MoE, which uses sensitivity-aware routing alignment and asymmetric capacity regularization, achieving consistent improvements in average and last accuracy over state-of-the-art methods on four benchmarks.
Class-incremental learning (CIL) requires models to learn new classes sequentially while preserving prior knowledge. Recently, approaches that combine pre-trained models with mixture-of-experts (MoE) have received increasing attention in CIL: they typically expand experts during learning and employ a router to assign weights across experts. However, existing MoE methods often overlook routing drift induced by expert expansion. Once new experts are introduced, the router may reassign samples from earlier classes to newly added experts, thereby perturbing previously established expert compositions and causing interference even when old experts remain frozen. We argue that expandable MoE in CIL requires two complementary properties: stable old-class routing for knowledge preservation and sufficient capacity utilization for new-class adaptation. To this end, we propose Stable Routing for MoE (StaR-MoE), a routing-level framework for expandable MoE in CIL. By incorporating sensitivity-aware routing alignment, StaR-MoE aligns current old-class routing behavior with historical routing distributions through sensitivity-guided constraints. Complementarily, StaR-MoE introduces asymmetric capacity regularization to encourage effective utilization of the expanded expert pool without compromising class-specific routing specialization. Extensive experiments across four standard CIL benchmarks demonstrate that StaR-MoE consistently improves both average and last accuracy over state-of-the-art methods, highlighting the importance of stable routing.