AdaCap: An Adaptive Contrastive Approach for Small-Data Neural Networks
This addresses the challenge of making neural networks more competitive with tree-based models in small-data scenarios, which is an incremental but practical advancement for machine learning practitioners.
The paper tackles the problem of neural networks underperforming on small tabular datasets by introducing AdaCap, a training scheme that improves performance, particularly for residual models, across 85 real-world regression datasets with statistically significant gains.
Neural networks struggle on small tabular datasets, where tree-based models remain dominant. We introduce Adaptive Contrastive Approach (AdaCap), a training scheme that combines a permutation-based contrastive loss with a Tikhonov-based closed-form output mapping. Across 85 real-world regression datasets and multiple architectures, AdaCap yields consistent and statistically significant improvements in the small-sample regime, particularly for residual models. A meta-predictor trained on dataset characteristics (size, skewness, noise) accurately anticipates when AdaCap is beneficial. These results show that AdaCap acts as a targeted regularization mechanism, strengthening neural networks precisely where they are most fragile. All results and code are publicly available at https://github.com/BrunoBelucci/adacap.