Geometric Regularization in Mixture-of-Experts: The Disconnect Between Weights and Activations
This work addresses the problem of unreliable regularization techniques for expert specialization in MoE models, which is incremental as it critiques an existing approach without proposing a new solution.
The study investigated the effectiveness of geometric regularization, specifically orthogonality loss, in promoting expert diversity in Mixture-of-Experts models, finding that it fails to reduce weight-space overlap (increasing by up to 114%), leaves activation-space overlap high (~0.6), and yields inconsistent performance changes across datasets.
Mixture-of-Experts (MoE) models achieve efficiency through sparse activation, but the role of geometric regularization in expert specialization remains unclear. We apply orthogonality loss to enforce expert diversity and find it fails on multiple fronts: it does not reduce weight-space overlap (MSO actually increases by up to 114%), activation-space overlap remains high (~0.6) regardless of regularization, and effects on performance are inconsistent -- marginal improvement on WikiText-103 (-0.9%), slight degradation on TinyStories (+0.9%), and highly variable results on PTB (std > 1.0). Our analysis across 7 regularization strengths reveals no significant correlation (r = -0.293, p = 0.523) between weight and activation orthogonality. These findings demonstrate that weight-space regularization neither achieves its geometric goal nor reliably improves performance, making it unsuitable for MoE diversity.