Squeezing Lemons with Hammers: An Evaluation of AutoML and Tabular Deep Learning for Data-Scarce Classification Applications
This provides practical guidance for practitioners in industry verticals dealing with small tabular datasets, though it is incremental as it compares existing methods.
The paper tackled the problem of selecting the best machine learning approach for data-scarce tabular classification, finding that L2-regularized logistic regression performs similarly to state-of-the-art AutoML and deep learning methods on 44 datasets with ≤500 samples.
Many industry verticals are confronted with small-sized tabular data. In this low-data regime, it is currently unclear whether the best performance can be expected from simple baselines, or more complex machine learning approaches that leverage meta-learning and ensembling. On 44 tabular classification datasets with sample sizes $\leq$ 500, we find that L2-regularized logistic regression performs similar to state-of-the-art automated machine learning (AutoML) frameworks (AutoPrognosis, AutoGluon) and off-the-shelf deep neural networks (TabPFN, HyperFast) on the majority of the benchmark datasets. We therefore recommend to consider logistic regression as the first choice for data-scarce applications with tabular data and provide practitioners with best practices for further method selection.