HyperTab: Hypernetwork Approach for Deep Learning on Small Tabular Datasets
This addresses the problem of improving deep learning performance for practitioners working with small tabular datasets, where tree-based ensembles typically dominate, though it is incremental as it builds on existing hypernetwork and ensemble techniques.
The paper tackles the challenge of deep learning underperforming shallow methods on small tabular datasets by introducing HyperTab, a hypernetwork-based approach that combines Random Forests and neural networks to generate specialized ensembles, resulting in consistent outperformance on small data with statistical significance and comparable performance on larger datasets.
Deep learning has achieved impressive performance in many domains, such as computer vision and natural language processing, but its advantage over classical shallow methods on tabular datasets remains questionable. It is especially challenging to surpass the performance of tree-like ensembles, such as XGBoost or Random Forests, on small-sized datasets (less than 1k samples). To tackle this challenge, we introduce HyperTab, a hypernetwork-based approach to solving small sample problems on tabular datasets. By combining the advantages of Random Forests and neural networks, HyperTab generates an ensemble of neural networks, where each target model is specialized to process a specific lower-dimensional view of the data. Since each view plays the role of data augmentation, we virtually increase the number of training samples while keeping the number of trainable parameters unchanged, which prevents model overfitting. We evaluated HyperTab on more than 40 tabular datasets of a varying number of samples and domains of origin, and compared its performance with shallow and deep learning models representing the current state-of-the-art. We show that HyperTab consistently outranks other methods on small data (with a statistically significant difference) and scores comparable to them on larger datasets. We make a python package with the code available to download at https://pypi.org/project/hypertab/