Horseshoe Mixtures-of-Experts (HS-MoE)
This addresses the need for efficient and sparse expert usage in large-scale models like mixture-of-experts layers in large language models, where only a few experts are activated per token out of many.
The paper tackles the problem of sparse expert selection in mixture-of-experts architectures by proposing a Bayesian framework called Horseshoe Mixtures-of-Experts (HS-MoE), which combines horseshoe priors with input-dependent gating to achieve data-adaptive sparsity, and introduces a particle learning algorithm for sequential inference.
Horseshoe mixtures-of-experts (HS-MoE) models provide a Bayesian framework for sparse expert selection in mixture-of-experts architectures. We combine the horseshoe prior's adaptive global-local shrinkage with input-dependent gating, yielding data-adaptive sparsity in expert usage. Our primary methodological contribution is a particle learning algorithm for sequential inference, in which the filter is propagated forward in time while tracking only sufficient statistics. We also discuss how HS-MoE relates to modern mixture-of-experts layers in large language models, which are deployed under extreme sparsity constraints (e.g., activating a small number of experts per token out of a large pool).