Soft Gradient Boosting with Learnable Feature Transforms for Sequential Regression
This work addresses regression challenges for domains with limited data and high dimensionality, though it appears incremental as it builds on existing gradient boosting methods.
The authors tackled the problem of sequential regression in high-dimensional, data-scarce scenarios by proposing a soft gradient boosting framework with learnable linear feature transforms, resulting in effective and efficient performance increases as demonstrated on synthetic and real-world datasets.
We propose a soft gradient boosting framework for sequential regression that embeds a learnable linear feature transform within the boosting procedure. At each boosting iteration, we train a soft decision tree and learn a linear input feature transform Q together. This approach is particularly advantageous in high-dimensional, data-scarce scenarios, as it discovers the most relevant input representations while boosting. We demonstrate, using both synthetic and real-world datasets, that our method effectively and efficiently increases the performance by an end-to-end optimization of feature selection/transform and boosting while avoiding overfitting. We also extend our algorithm to differentiable non-linear transforms if overfitting is not a problem. To support reproducibility and future work, we share our code publicly.