In-context learning of evolving data streams with tabular foundational models
This addresses adaptive learning in dynamic environments for data stream mining, representing a significant paradigm shift from traditional incremental ensembles.
The paper tackles the problem of supervised classification in non-stationary data streams by proposing a method that uses tabular foundational models with in-context learning, demonstrating that TabPFN with a sliding memory strategy consistently outperforms ensembles of Hoeffding trees across benchmarks.
State-of-the-art data stream mining in supervised classification has traditionally relied on ensembles of incremental decision trees. However, the emergence of large tabular models, i.e., transformers designed for structured numerical data, marks a significant paradigm shift. These models move beyond traditional weight updates, instead employing in-context learning through prompt tuning. By using on-the-fly sketches to summarize unbounded streaming data, one can feed this information into a pre-trained model for efficient processing. This work bridges advancements from both areas, highlighting how transformers' implicit meta-learning abilities, pre-training on drifting natural data, and reliance on context optimization directly address the core challenges of adaptive learning in dynamic environments. Exploring real-time model adaptation, this research demonstrates that TabPFN, coupled with a simple sliding memory strategy, consistently outperforms ensembles of Hoeffding trees across all non-stationary benchmarks. Several promising research directions are outlined in the paper. The authors urge the community to explore these ideas, offering valuable opportunities to advance in-context stream learning.