Boosting with the Logistic Loss is Consistent
This work addresses the need for robust theoretical foundations in boosting algorithms for machine learning practitioners, though it appears incremental as it extends existing AdaBoost analysis to different loss functions.
The paper tackles the problem of providing theoretical guarantees for AdaBoost variants using logistic-like losses, showing that these algorithms achieve statistical consistency and exhibit distribution-dependent convergence properties in both separable and nonseparable cases.
This manuscript provides optimization guarantees, generalization bounds, and statistical consistency results for AdaBoost variants which replace the exponential loss with the logistic and similar losses (specifically, twice differentiable convex losses which are Lipschitz and tend to zero on one side). The heart of the analysis is to show that, in lieu of explicit regularization and constraints, the structure of the problem is fairly rigidly controlled by the source distribution itself. The first control of this type is in the separable case, where a distribution-dependent relaxed weak learning rate induces speedy convergence with high probability over any sample. Otherwise, in the nonseparable case, the convex surrogate risk itself exhibits distribution-dependent levels of curvature, and consequently the algorithm's output has small norm with high probability.