Functional Gradient Boosting based on Residual Network Perception
This work addresses the need for more powerful gradient boosting techniques in data analysis, offering a novel approach that combines ResNet insights, though it appears incremental as it builds on existing ResNet and gradient boosting frameworks.
The paper tackled the problem of improving gradient boosting methods by formalizing a functional gradient perspective based on Residual Networks (ResNets) and proposed ResFGB for classification, achieving superior performance over state-of-the-art methods like LightGBM in experiments.
Residual Networks (ResNets) have become state-of-the-art models in deep learning and several theoretical studies have been devoted to understanding why ResNet works so well. One attractive viewpoint on ResNet is that it is optimizing the risk in a functional space by combining an ensemble of effective features. In this paper, we adopt this viewpoint to construct a new gradient boosting method, which is known to be very powerful in data analysis. To do so, we formalize the gradient boosting perspective of ResNet mathematically using the notion of functional gradients and propose a new method called ResFGB for classification tasks by leveraging ResNet perception. Two types of generalization guarantees are provided from the optimization perspective: one is the margin bound and the other is the expected risk bound by the sample-splitting technique. Experimental results show superior performance of the proposed method over state-of-the-art methods such as LightGBM.