SGLB: Stochastic Gradient Langevin Boosting
This addresses a limitation in gradient boosting for practitioners dealing with multimodal loss functions, though it is an incremental improvement over existing methods.
The paper tackled the problem of gradient boosting algorithms getting stuck in local optima for multimodal loss functions by introducing Stochastic Gradient Langevin Boosting (SGLB), which theoretically guarantees global convergence and empirically outperforms classic gradient boosting on classification tasks with 0-1 loss.
This paper introduces Stochastic Gradient Langevin Boosting (SGLB) - a powerful and efficient machine learning framework that may deal with a wide range of loss functions and has provable generalization guarantees. The method is based on a special form of the Langevin diffusion equation specifically designed for gradient boosting. This allows us to theoretically guarantee the global convergence even for multimodal loss functions, while standard gradient boosting algorithms can guarantee only local optimum. We also empirically show that SGLB outperforms classic gradient boosting when applied to classification tasks with 0-1 loss function, which is known to be multimodal.