Tabular Data: Deep Learning is Not All You Need
This addresses the problem of efficient and effective model choice for data scientists working with tabular data, but it is incremental as it builds on existing comparisons.
This paper tackles the problem of model selection for tabular data by rigorously comparing deep learning models to XGBoost, finding that XGBoost outperforms deep models across datasets and requires less tuning, while an ensemble of both performs better than XGBoost alone.
A key element in solving real-life data science problems is selecting the types of models to use. Tree ensemble models (such as XGBoost) are usually recommended for classification and regression problems with tabular data. However, several deep learning models for tabular data have recently been proposed, claiming to outperform XGBoost for some use cases. This paper explores whether these deep models should be a recommended option for tabular data by rigorously comparing the new deep models to XGBoost on various datasets. In addition to systematically comparing their performance, we consider the tuning and computation they require. Our study shows that XGBoost outperforms these deep models across the datasets, including the datasets used in the papers that proposed the deep models. We also demonstrate that XGBoost requires much less tuning. On the positive side, we show that an ensemble of deep models and XGBoost performs better on these datasets than XGBoost alone.