Probabilistic Pretraining for Neural Regression
This work addresses the problem of enhancing predictive performance in probabilistic regression for machine learning practitioners, though it appears incremental as it builds on existing transfer learning concepts.
The paper tackles the underexplored problem of transfer learning for probabilistic regression by introducing NIAQUE, a model that uses permutation invariance for any-quantile estimation, and demonstrates that pre-training on diverse datasets and fine-tuning improves performance on specific tasks, with effectiveness shown in Kaggle competitions against strong baselines.
Transfer learning for probabilistic regression remains underexplored. This work closes this gap by introducing NIAQUE, Neural Interpretable Any-Quantile Estimation, a new model designed for transfer learning in probabilistic regression through permutation invariance. We demonstrate that pre-training NIAQUE directly on diverse downstream regression datasets and fine-tuning it on a specific target dataset enhances performance on individual regression tasks, showcasing the positive impact of probabilistic transfer learning. Furthermore, we highlight the effectiveness of NIAQUE in Kaggle competitions against strong baselines involving tree-based models and recent neural foundation models TabPFN and TabDPT. The findings highlight NIAQUE's efficacy as a robust and scalable framework for probabilistic regression, leveraging transfer learning to enhance predictive performance.