Variational Routing: A Scalable Bayesian Framework for Calibrated Mixture-of-Experts Transformers
This addresses the need for calibrated uncertainty in foundation models for responsible deployment, offering a scalable solution with significant performance gains.
The paper tackles the problem of uncertainty quantification in large-scale foundation models by introducing Variational Mixture-of-Experts Routing (VMoER), a Bayesian framework for MoE layers, which improves routing stability by 38%, reduces calibration error by 94%, and increases out-of-distribution AUROC by 12% with minimal computational overhead.
Foundation models are increasingly being deployed in contexts where understanding the uncertainty of their outputs is critical to ensuring responsible deployment. While Bayesian methods offer a principled approach to uncertainty quantification, their computational overhead renders their use impractical for training or inference at foundation model scale. State-of-the-art models achieve parameter counts in the trillions through carefully engineered sparsity including Mixture-of-Experts (MoE) layers. In this work, we demonstrate calibrated uncertainty at scale by introducing Variational Mixture-of-Experts Routing (VMoER), a structured Bayesian approach for modelling uncertainty in MoE layers. VMoER confines Bayesian inference to the expert-selection stage which is typically done by a deterministic routing network. We instantiate VMoER using two inference strategies: amortised variational inference over routing logits and inferring a temperature parameter for stochastic expert selection. Across tested foundation models, VMoER improves routing stability under noise by 38\%, reduces calibration error by 94\%, and increases out-of-distribution AUROC by 12\%, while incurring less than 1\% additional FLOPs. These results suggest VMoER offers a scalable path toward robust and uncertainty-aware foundation models.