Rethinking Multinomial Logistic Mixture of Experts with Sigmoid Gating Function
This work addresses theoretical gaps in mixture-of-experts models for classification, offering improved efficiency for machine learning practitioners, though it is incremental as it builds on existing sigmoid gate research.
The paper tackles the lack of theoretical understanding of sigmoid-gated mixture-of-experts models in classification settings, showing that a modified sigmoid gate has lower sample complexity than softmax for parameter and expert estimation, and proposing a Euclidean score to improve sample complexity from exponential to polynomial order.
The sigmoid gate in mixture-of-experts (MoE) models has been empirically shown to outperform the softmax gate across several tasks, ranging from approximating feed-forward networks to language modeling. Additionally, recent efforts have demonstrated that the sigmoid gate is provably more sample-efficient than its softmax counterpart under regression settings. Nevertheless, there are three notable concerns that have not been addressed in the literature, namely (i) the benefits of the sigmoid gate have not been established under classification settings; (ii) existing sigmoid-gated MoE models may not converge to their ground-truth; and (iii) the effects of a temperature parameter in the sigmoid gate remain theoretically underexplored. To tackle these open problems, we perform a comprehensive analysis of multinomial logistic MoE equipped with a modified sigmoid gate to ensure model convergence. Our results indicate that the sigmoid gate exhibits a lower sample complexity than the softmax gate for both parameter and expert estimation. Furthermore, we find that incorporating a temperature into the sigmoid gate leads to a sample complexity of exponential order due to an intrinsic interaction between the temperature and gating parameters. To overcome this issue, we propose replacing the vanilla inner product score in the gating function with a Euclidean score that effectively removes that interaction, thereby substantially improving the sample complexity to a polynomial order.