Early Stopping Tabular In-Context Learning
This addresses efficiency issues for users of tabular in-context learning, though it is incremental as it builds on existing methods with a speed optimization.
The paper tackles the high inference-time costs of tabular foundation models in in-context learning by proposing early stopping, which accelerates inference by up to 1.3x on small tasks and up to 2.2x on larger tasks with minimal performance loss.
Tabular foundation models have shown strong performance across various tabular learning tasks via in-context learning, offering robust generalization without any downstream finetuning. However, their inference-time costs remain high, particularly for larger datasets. To address this, we propose early-stopping the in-context learning process. We achieve this by dynamically evaluating whether to stop in-context learning after each Transformer encoder layer. Once stopped, we decode the embedding using a pre-trained layer-wise decoder. Experiments across 34 small classification tasks size show that early stopping in-context learning accelerates inference by up to x1.3 with negligible degradation in predictive performance. To assess scalability, we further evaluate our method on five larger classification tasks, achieving speedups of up to x2.2. Our results demonstrate the potential of early exiting as an effective and practical strategy for improving the efficiency of tabular in-context learning.