Risk and parameter convergence of logistic regression
This provides theoretical insights into convergence behavior for logistic regression, which is incremental but clarifies parameter dynamics in machine learning optimization.
The paper analyzes gradient descent applied to logistic regression, showing that iterates converge directionally to a unique ray defined by the maximum margin predictor of a separable data subset at a rate of O(ln ln t / ln t), and recover an offset at a rate of O((ln t)^2 / sqrt(t)).
Gradient descent, when applied to the task of logistic regression, outputs iterates which are biased to follow a unique ray defined by the data. The direction of this ray is the maximum margin predictor of a maximal linearly separable subset of the data; the gradient descent iterates converge to this ray in direction at the rate $\mathcal{O}(\ln\ln t / \ln t)$. The ray does not pass through the origin in general, and its offset is the bounded global optimum of the risk over the remaining data; gradient descent recovers this offset at a rate $\mathcal{O}((\ln t)^2 / \sqrt{t})$.