Statistical Foundations of Prior-Data Fitted Networks
This provides foundational insights for designing architectures in machine learning, though it is incremental as it builds on existing PFN concepts.
The paper tackles the theoretical understanding of prior-data fitted networks (PFNs), a new paradigm where a fixed model is pre-trained on simulated tasks and used for in-context inference, showing that PFNs achieve state-of-the-art performance on tasks similar in size to pre-training and improve with larger inference data. It establishes a frequentistic interpretation explaining that variance vanishes with low sensitivity to training samples, but bias reduction requires localization around test features, which current transformer architectures do not fully ensure.
Prior-data fitted networks (PFNs) were recently proposed as a new paradigm for machine learning. Instead of training the network to an observed training set, a fixed model is pre-trained offline on small, simulated training sets from a variety of tasks. The pre-trained model is then used to infer class probabilities in-context on fresh training sets with arbitrary size and distribution. Empirically, PFNs achieve state-of-the-art performance on tasks with similar size to the ones used in pre-training. Surprisingly, their accuracy further improves when passed larger data sets during inference. This article establishes a theoretical foundation for PFNs and illuminates the statistical mechanisms governing their behavior. While PFNs are motivated by Bayesian ideas, a purely frequentistic interpretation of PFNs as pre-tuned, but untrained predictors explains their behavior. A predictor's variance vanishes if its sensitivity to individual training samples does and the bias vanishes only if it is appropriately localized around the test feature. The transformer architecture used in current PFN implementations ensures only the former. These findings shall prove useful for designing architectures with favorable empirical behavior.