A similarity-based Bayesian mixture-of-experts model
This work provides an improved regression model for practitioners dealing with high-dimensional data and complex input-output relationships, where predictive distributions may be skewed or multimodal.
This paper introduces a new nonparametric mixture-of-experts model for multivariate regression, which uses similarities to observed data points to generate predictive distributions. The model demonstrates clear advantages over competitor models on five datasets, particularly for high-dimensional inputs, in terms of validation metrics and visual inspection.
We present a new nonparametric mixture-of-experts model for multivariate regression problems, inspired by the probabilistic k-nearest neighbors algorithm. Using a conditionally specified model, predictions for out-of-sample inputs are based on similarities to each observed data point, yielding predictive distributions represented by Gaussian mixtures. Posterior inference is performed on the parameters of the mixture components as well as the distance metric using a mean-field variational Bayes algorithm accompanied with a stochastic gradient-based optimization procedure. The proposed method is especially advantageous in settings where inputs are of relatively high dimension in comparison to the data size, where input-output relationships are complex, and where predictive distributions may be skewed or multimodal. Computational studies on five datasets, of which two are synthetically generated, illustrate clear advantages of our mixture-of-experts method for high-dimensional inputs, outperforming competitor models both in terms of validation metrics and visual inspection.