A Universal Approximation Theorem for Mixture of Experts Models
This provides theoretical justification for using MoE models in general function approximation tasks, though it is an incremental theoretical extension of existing results.
The authors proved that mixture of experts (MoE) models can approximate any continuous function over arbitrary compact domains, establishing a universal approximation theorem for MoE architectures.
The mixture of experts (MoE) model is a popular neural network architecture for nonlinear regression and classification. The class of MoE mean functions is known to be uniformly convergent to any unknown target function, assuming that the target function is from Sobolev space that is sufficiently differentiable and that the domain of estimation is a compact unit hypercube. We provide an alternative result, which shows that the class of MoE mean functions is dense in the class of all continuous functions over arbitrary compact domains of estimation. Our result can be viewed as a universal approximation theorem for MoE models.