Prior-Fitted Networks Scale to Larger Datasets When Treated as Weak Learners
This work addresses the problem of efficiently scaling PFNs for tabular classification on large datasets, which is incremental as it builds on existing PFN methods.
The paper tackles the limitations of Prior-Fitted Networks (PFNs) on large datasets, such as high memory and computational costs, by proposing BoostPFN, which outperforms standard PFNs and accelerates training compared to baselines like GBDTs and deep learning methods, maintaining performance up to 50x the pre-training size.
Prior-Fitted Networks (PFNs) have recently been proposed to efficiently perform tabular classification tasks. Although they achieve good performance on small datasets, they encounter limitations with larger datasets. These limitations include significant memory consumption and increased computational complexity, primarily due to the impracticality of incorporating all training samples as inputs within these networks. To address these challenges, we investigate the fitting assumption for PFNs and input samples. Building on this understanding, we propose \textit{BoostPFN} designed to enhance the performance of these networks, especially for large-scale datasets. We also theoretically validate the convergence of BoostPFN and our empirical results demonstrate that the BoostPFN method can outperform standard PFNs with the same size of training samples in large datasets and achieve a significant acceleration in training times compared to other established baselines in the field, including widely-used Gradient Boosting Decision Trees (GBDTs), deep learning methods and AutoML systems. High performance is maintained for up to 50x of the pre-training size of PFNs, substantially extending the limit of training samples. Through this work, we address the challenges of efficiently handling large datasets via PFN-based models, paving the way for faster and more effective tabular data classification training and prediction process. Code is available at Github.