What exactly has TabPFN learned to do?
This work provides insights for developing and evaluating prior-data fitted networks in tabular data, but it is incremental as it focuses on understanding an existing model without introducing new methods or broad improvements.
The paper investigates the learned inductive biases of TabPFN, a Transformer model pretrained for in-context learning on tabular classification, by analyzing its function approximations as a black-box generator, revealing mixed brilliant and baffling behaviors.
TabPFN [Hollmann et al., 2023], a Transformer model pretrained to perform in-context learning on fresh tabular classification problems, was presented at the last ICLR conference. To better understand its behavior, we treat it as a black-box function approximator generator and observe its generated function approximations on a varied selection of training datasets. Exploring its learned inductive biases in this manner, we observe behavior that is at turns either brilliant or baffling. We conclude this post with thoughts on how these results might inform the development, evaluation, and application of prior-data fitted networks (PFNs) in the future.