Random Feature Representation Boosting
This work addresses the need for more efficient and effective neural network methods in machine learning, particularly for tabular data applications, though it appears incremental as it builds on existing RFNN and boosting frameworks.
The paper tackles the problem of improving deep residual random feature neural networks by introducing Random Feature Representation Boosting (RFRBoost), which uses boosting theory to learn functional gradients with random features, resulting in significant performance gains over existing methods like RFNNs and MLP ResNets on tabular datasets in small- to medium-scale regimes.
We introduce Random Feature Representation Boosting (RFRBoost), a novel method for constructing deep residual random feature neural networks (RFNNs) using boosting theory. RFRBoost uses random features at each layer to learn the functional gradient of the network representation, enhancing performance while preserving the convex optimization benefits of RFNNs. In the case of MSE loss, we obtain closed-form solutions to greedy layer-wise boosting with random features. For general loss functions, we show that fitting random feature residual blocks reduces to solving a quadratically constrained least squares problem. Through extensive numerical experiments on tabular datasets for both regression and classification, we show that RFRBoost significantly outperforms RFNNs and end-to-end trained MLP ResNets in the small- to medium-scale regime where RFNNs are typically applied. Moreover, RFRBoost offers substantial computational benefits, and theoretical guarantees stemming from boosting theory.