Hierarchical Mixtures-of-Experts for Exponential Family Regression Models with Generalized Linear Mean Functions: A Survey of Approximation and Consistency Results
This provides theoretical guarantees for using HME models in statistical regression, which is incremental as it extends existing approximation and consistency results to a broader class of models.
The paper tackles the problem of approximating true exponential family regression models using hierarchical mixtures-of-experts (HME) models, showing that HME can achieve approximation rates of O(m^{-2/s}) in L_p norm and O(m^{-4/s}) in Kullback-Leibler divergence, and that likelihood-based inference is consistent with mean square error converging to zero as sample size and number of experts increase.
We investigate a class of hierarchical mixtures-of-experts (HME) models where exponential family regression models with generalized linear mean functions of the form psi(ga+fx^Tfgb) are mixed. Here psi(...) is the inverse link function. Suppose the true response y follows an exponential family regression model with mean function belonging to a class of smooth functions of the form psi(h(fx)) where h(...)in W_2^infty (a Sobolev class over [0,1]^{s}). It is shown that the HME probability density functions can approximate the true density, at a rate of O(m^{-2/s}) in L_p norm, and at a rate of O(m^{-4/s}) in Kullback-Leibler divergence. These rates can be achieved within the family of HME structures with no more than s-layers, where s is the dimension of the predictor fx. It is also shown that likelihood-based inference based on HME is consistent in recovering the truth, in the sense that as the sample size n and the number of experts m both increase, the mean square error of the predicted mean response goes to zero. Conditions for such results to hold are stated and discussed.