A Performance-Driven Benchmark for Feature Selection in Tabular Deep Learning
This work addresses a practical problem for data scientists and researchers in tabular deep learning by providing a more realistic benchmark and improved feature selection method, though it is incremental as it builds on existing techniques.
The authors tackled the lack of performance-driven benchmarks for feature selection in tabular deep learning by constructing a challenging benchmark using real datasets and neural networks, and proposed a new input-gradient-based method that outperforms classical approaches, achieving better results on tasks like selecting from corrupted or second-order features.
Academic tabular benchmarks often contain small sets of curated features. In contrast, data scientists typically collect as many features as possible into their datasets, and even engineer new features from existing ones. To prevent overfitting in subsequent downstream modeling, practitioners commonly use automated feature selection methods that identify a reduced subset of informative features. Existing benchmarks for tabular feature selection consider classical downstream models, toy synthetic datasets, or do not evaluate feature selectors on the basis of downstream performance. Motivated by the increasing popularity of tabular deep learning, we construct a challenging feature selection benchmark evaluated on downstream neural networks including transformers, using real datasets and multiple methods for generating extraneous features. We also propose an input-gradient-based analogue of Lasso for neural networks that outperforms classical feature selection methods on challenging problems such as selecting from corrupted or second-order features.