TabGemma: Text-Based Tabular ICL via LLM using Continued Pretraining and Retrieval
This work addresses the challenge of using LLMs for tabular data tasks, which is important for data science and machine learning applications, though it is incremental in improving existing methods.
The authors tackled the problem of adapting large language models for tabular prediction with mixed data types by introducing TabGemma, which addresses numeric tokenization and context size limitations, achieving state-of-the-art results in classification across data regimes and competitive performance in regression at small sample sizes.
We study LLMs for tabular prediction with mixed text, numeric, and categorical fields. We introduce TabGemma, a schema-agnostic in-context learner that treats rows as sequences and tackles two practical hurdles when adapting pretrained LLMs for tabular predictions: unstable numeric tokenization and limited context size. We propose to canonicalize numbers via signed scientific notation and continue pretraining of a 12B Gemma 3 model with a target imputation objective using a large-scale real world dataset. For inference, we use a compact n-gram-based retrieval to select informative exemplars that fit within a 128k-token window. On semantically rich benchmarks, TabGemma establishes a new state of the art on classification across low- and high-data regimes and improves monotonically with more context rows. For regression, it is competitive at small sample sizes but trails conventional approaches as data grows. Our results show that LLMs can be effective tabular in-context learners on highly semantic tasks when paired with dedicated numeric handling and context retrieval, while motivating further advances in numeric modeling and long-context scaling.