Better by Default: Strong Pre-Tuned MLPs and Boosted Trees on Tabular Data
This work addresses the problem of reducing the need for extensive hyperparameter tuning in tabular data analysis for machine learning practitioners, offering incremental improvements to existing methods.
The paper tackles the challenge of improving performance on tabular data by introducing RealMLP, an enhanced multilayer perceptron, and strong default parameters for both RealMLP and gradient-boosted decision trees (GBDTs), showing that RealMLP offers a favorable time-accuracy tradeoff compared to other neural baselines and is competitive with GBDTs on benchmarks with 1K-500K samples.
For classification and regression on tabular data, the dominance of gradient-boosted decision trees (GBDTs) has recently been challenged by often much slower deep learning methods with extensive hyperparameter tuning. We address this discrepancy by introducing (a) RealMLP, an improved multilayer perceptron (MLP), and (b) strong meta-tuned default parameters for GBDTs and RealMLP. We tune RealMLP and the default parameters on a meta-train benchmark with 118 datasets and compare them to hyperparameter-optimized versions on a disjoint meta-test benchmark with 90 datasets, as well as the GBDT-friendly benchmark by Grinsztajn et al. (2022). Our benchmark results on medium-to-large tabular datasets (1K--500K samples) show that RealMLP offers a favorable time-accuracy tradeoff compared to other neural baselines and is competitive with GBDTs in terms of benchmark scores. Moreover, a combination of RealMLP and GBDTs with improved default parameters can achieve excellent results without hyperparameter tuning. Finally, we demonstrate that some of RealMLP's improvements can also considerably improve the performance of TabR with default parameters.