A Flexible Empirical Bayes Approach to Generalized Linear Models, with Applications to Sparse Logistic Regression
This work provides a unified framework for Bayesian generalized linear models, potentially benefiting researchers and practitioners in machine learning and statistics by simplifying model fitting across exponential family distributions.
The authors tackled the problem of fitting Bayesian generalized linear models by introducing a flexible empirical Bayes approach that uses a novel mean-field variational inference method, which is tuning-free and scalable. They demonstrated superior predictive performance in sparse logistic regression compared to existing methods.
We introduce a flexible empirical Bayes approach for fitting Bayesian generalized linear models. Specifically, we adopt a novel mean-field variational inference (VI) method and the prior is estimated within the VI algorithm, making the method tuning-free. Unlike traditional VI methods that optimize the posterior density function, our approach directly optimizes the posterior mean and prior parameters. This formulation reduces the number of parameters to optimize and enables the use of scalable algorithms such as L-BFGS and stochastic gradient descent. Furthermore, our method automatically determines the optimal posterior based on the prior and likelihood, distinguishing it from existing VI methods that often assume a Gaussian variational. Our approach represents a unified framework applicable to a wide range of exponential family distributions, removing the need to develop unique VI methods for each combination of likelihood and prior distributions. We apply the framework to solve sparse logistic regression and demonstrate the superior predictive performance of our method in extensive numerical studies, by comparing it to prevalent sparse logistic regression approaches.