Expectation-maximization for logistic regression
This work provides a new algorithmic perspective for logistic regression, which is incremental as it reinterprets existing methods and extends them with modest improvements.
The paper tackles the problem of logistic regression by developing a family of expectation-maximization (EM) algorithms, showing that a variational-Bayes approach can be reinterpreted as a true EM algorithm, and generalizing it to sparsity-promoting priors and an online method that outperforms stochastic-gradient descent in collinear situations.
We present a family of expectation-maximization (EM) algorithms for binary and negative-binomial logistic regression, drawing a sharp connection with the variational-Bayes algorithm of Jaakkola and Jordan (2000). Indeed, our results allow a version of this variational-Bayes approach to be re-interpreted as a true EM algorithm. We study several interesting features of the algorithm, and of this previously unrecognized connection with variational Bayes. We also generalize the approach to sparsity-promoting priors, and to an online method whose convergence properties are easily established. This latter method compares favorably with stochastic-gradient descent in situations with marked collinearity.