Gradient Boosting Mapping for Dimensionality Reduction and Feature Extraction
This addresses the need for efficient and interpretable feature transformation in supervised learning, though it appears incremental as it builds on existing gradient boosting and dimensionality reduction concepts.
The paper tackles the problem of supervised dimensionality reduction and feature extraction by proposing Gradient Boosting Mapping (GBMAP), which uses weak learners to create embeddings that improve features for supervised tasks, making linear models competitive with state-of-the-art methods and enabling fast processing for large datasets.
A fundamental problem in supervised learning is to find a good set of features or distance measures. If the new set of features is of lower dimensionality and can be obtained by a simple transformation of the original data, they can make the model understandable, reduce overfitting, and even help to detect distribution drift. We propose a supervised dimensionality reduction method Gradient Boosting Mapping (GBMAP), where the outputs of weak learners -- defined as one-layer perceptrons -- define the embedding. We show that the embedding coordinates provide better features for the supervised learning task, making simple linear models competitive with the state-of-the-art regressors and classifiers. We also use the embedding to find a principled distance measure between points. The features and distance measures automatically ignore directions irrelevant to the supervised learning task. We also show that we can reliably detect out-of-distribution data points with potentially large regression or classification errors. GBMAP is fast and works in seconds for dataset of million data points or hundreds of features. As a bonus, GBMAP provides a regression and classification performance comparable to the state-of-the-art supervised learning methods.