State-Space Models for Tabular Prior-Data Fitted Networks
This work addresses efficiency issues in tabular data processing for machine learning practitioners, but it is incremental as it adapts an existing model to a new architecture.
The paper tackled the quadratic complexity of Transformers in tabular data models by exploring Hydra, a bidirectional state-space model, to reduce order-dependence and maintain efficiency, achieving predictive performance competitive with the original TabPFN model.
Recent advancements in foundation models for tabular data, such as TabPFN, demonstrated that pretrained Transformer architectures can approximate Bayesian inference with high predictive performance. However, Transformers suffer from quadratic complexity with respect to sequence length, motivating the exploration of more efficient sequence models. In this work, we investigate the potential of using Hydra, a bidirectional linear-time structured state space model (SSM), as an alternative to Transformers in TabPFN. A key challenge lies in SSM's inherent sensitivity to the order of input tokens - an undesirable property for tabular datasets where the row order is semantically meaningless. We investigate to what extent a bidirectional approach can preserve efficiency and enable symmetric context aggregation. Our experiments show that this approach reduces the order-dependence, achieving predictive performance competitive to the original TabPFN model.