Spectral Manifold Regularization for Stable and Modular Routing in Deep MoE Architectures
This addresses the issue of stable and modular routing for building high-capacity, lifelong learning networks in deep learning, representing a novel method for a known bottleneck.
The paper tackles the problem of expert collapse in Mixture of Experts (MoE) architectures, which reduces model capacity and causes catastrophic interference, by proposing SR-MoE with spectral manifold regularization, resulting in maintained structural integrity with mean interference of -0.32% compared to accuracy drops up to 4.72% for traditional methods.
Mixture of Experts (MoE) architectures enable efficient scaling of neural networks but suffer from expert collapse, where routing converges to a few dominant experts. This reduces model capacity and causes catastrophic interference during adaptation. We propose the Spectrally-Regularized Mixture of Experts (SR-MoE), which imposes geometric constraints on the routing manifold to enforce structural modularity. Our method uses dual regularization: spectral norm constraints bound routing function Lipschitz continuity, while stable rank penalties preserve high-dimensional feature diversity in expert selection. We evaluate SR-MoE across architectural scales and dataset complexities using modular one-shot adaptation tasks. Results show that traditional linear gating fails with increasing depth (accuracy drops up to 4.72% due to expert entanglement), while SR-MoE maintains structural integrity (mean interference -0.32%). Our spectral constraints facilitate positive knowledge transfer, enabling localized expert updates without global performance decay. SR-MoE provides a general solution for building high-capacity, modular networks capable of stable lifelong learning.