Chunked TabPFN: Exact Training-Free In-Context Learning for Long-Context Tabular Data
This work addresses a bottleneck for researchers and practitioners using TabPFN in tabular data analysis, though it is incremental as it builds on an existing framework.
The paper tackles the problem of TabPFN's inability to handle long-context tabular data due to quadratic computational costs by introducing a tiled-block strategy for attention computation, enabling it to process long contexts without pre-processing and demonstrating effectiveness on the TabArena benchmark.
TabPFN v2 achieves better results than tree-based models on several tabular benchmarks, which is notable since tree-based models are usually the strongest choice for tabular data. However, it cannot handle more than 10K context tokens because transformers have quadratic computation and memory costs. Unlike existing approaches that rely on context compression, such as selecting representative samples via K-nearest neighbors (KNN), we introduce a tiled-block strategy to compute attention within the TabPFN framework. This design is compatible with standard GPU setups and, to the best of our knowledge, is the first to enable TabPFN to process long contexts without any pre-processing. We demonstrate the effectiveness of our approach on the standard TabArena benchmark, with code available at https://github.com/mrsergazinov/chunk_tabpfn.