Well-tuned Simple Nets Excel on Tabular Datasets
This addresses the challenge of applying deep learning to tabular data, offering a method that surpasses existing approaches, though it is incremental in its focus on regularization tuning.
The paper tackled the problem of improving neural network performance on tabular datasets by optimizing combinations of 13 regularization techniques for MLPs, resulting in well-regularized MLPs outperforming specialized neural architectures and traditional methods like XGBoost on 40 datasets.
Tabular datasets are the last "unconquered castle" for deep learning, with traditional ML methods like Gradient-Boosted Decision Trees still performing strongly even against recent specialized neural architectures. In this paper, we hypothesize that the key to boosting the performance of neural networks lies in rethinking the joint and simultaneous application of a large set of modern regularization techniques. As a result, we propose regularizing plain Multilayer Perceptron (MLP) networks by searching for the optimal combination/cocktail of 13 regularization techniques for each dataset using a joint optimization over the decision on which regularizers to apply and their subsidiary hyperparameters. We empirically assess the impact of these regularization cocktails for MLPs in a large-scale empirical study comprising 40 tabular datasets and demonstrate that (i) well-regularized plain MLPs significantly outperform recent state-of-the-art specialized neural network architectures, and (ii) they even outperform strong traditional ML methods, such as XGBoost.