Margins, Shrinkage, and Boosting
This work offers theoretical guarantees for boosting methods, addressing regularization in machine learning, but it is incremental as it builds on existing shrinkage techniques.
The paper shows that AdaBoost and its variants can produce approximate maximum margin classifiers by scaling step sizes with a small constant, providing guarantees for Friedman's shrinkage procedure in gradient boosting and other step sizes, with results applicable to exponential and logistic losses.
This manuscript shows that AdaBoost and its immediate variants can produce approximate maximum margin classifiers simply by scaling step size choices with a fixed small constant. In this way, when the unscaled step size is an optimal choice, these results provide guarantees for Friedman's empirically successful "shrinkage" procedure for gradient boosting (Friedman, 2000). Guarantees are also provided for a variety of other step sizes, affirming the intuition that increasingly regularized line searches provide improved margin guarantees. The results hold for the exponential loss and similar losses, most notably the logistic loss.