Statistical optimality and stability of tangent transform algorithms in logit models
This provides theoretical foundations for variational inference methods in logit models, addressing a gap for researchers in Bayesian statistics and machine learning, though it is incremental as it builds on existing convex duality approaches.
The authors tackled the lack of theoretical guarantees for variational inference in non-conjugate Bayesian models, specifically in logistic regression, by deriving non-asymptotic risk bounds for variational optima under mild conditions and proving local asymptotic stability for tangent transform algorithms without data assumptions.
A systematic approach to finding variational approximation in an otherwise intractable non-conjugate model is to exploit the general principle of convex duality by minorizing the marginal likelihood that renders the problem tractable. While such approaches are popular in the context of variational inference in non-conjugate Bayesian models, theoretical guarantees on statistical optimality and algorithmic convergence are lacking. Focusing on logistic regression models, we provide mild conditions on the data generating process to derive non-asymptotic upper bounds to the risk incurred by the variational optima. We demonstrate that these assumptions can be completely relaxed if one considers a slight variation of the algorithm by raising the likelihood to a fractional power. Next, we utilize the theory of dynamical systems to provide convergence guarantees for such algorithms in logistic and multinomial logit regression. In particular, we establish local asymptotic stability of the algorithm without any assumptions on the data-generating process. We explore a special case involving a semi-orthogonal design under which a global convergence is obtained. The theory is further illustrated using several numerical studies.