Geometry-Preserving Aggregation for Mixture-of-Experts Embedding Models
This addresses a geometry distortion problem in MoE embedding models for NLP tasks, offering an incremental improvement with a novel aggregation method.
The paper tackles the inconsistency between linear aggregation in Mixture-of-Experts embedding models and the hyperspherical geometry of expert representations, introducing Spherical Barycentric Aggregation (SBA) to preserve this geometry and showing consistent performance improvements on tasks from the Massive Text Embedding Benchmark with identical training cost and full stability.
Mixture-of-Experts (MoE) embedding models combine expert outputs using weighted linear summation, implicitly assuming a linear subspace structure in the embedding space. This assumption is shown to be inconsistent with the geometry of expert representations. Geometric analysis of a modern MoE embedding model reveals that expert outputs lie on a shared hyperspherical manifold characterized by tightly concentrated norms and substantial angular separation. Under this geometry, linear aggregation induces inward collapse toward the manifold interior, distorting vector magnitude and direction and reducing embedding comparability. To address this inconsistency, Spherical Barycentric Aggregation (SBA) is introduced as a geometry-preserving aggregation operator that separates radial and angular components to maintain hyperspherical structure while remaining fully compatible with existing routing mechanisms. Experiments on selected tasks from the Massive Text Embedding Benchmark (MTEB), including semantic similarity, clustering, and duplicate question detection, demonstrate consistent performance improvements with identical training cost and full stability. Additional geometric analyses confirm that SBA prevents aggregation-induced collapse and preserves hyperspherical consistency, highlighting the importance of geometry-aware aggregation in MoE embedding architectures.