BoostLLM: Boosting-inspired LLM Fine-tuning for Few-shot Tabular Classification
For practitioners of tabular classification in low-data regimes, BoostLLM offers a principled way to improve LLM fine-tuning performance, though it is incremental as it applies an existing paradigm (boosting) to a new domain (LLM fine-tuning).
BoostLLM adapts boosting to LLM fine-tuning for few-shot tabular classification, achieving consistent gains over standard fine-tuning and matching XGBoost across shot counts, with a 4B model outperforming GPT-4o-based methods.
Large language models (LLMs) have recently been adapted to tabular prediction by serializing structured features into natural language, but their performance in low-data regimes remains limited compared to gradient-boosted decision trees (GBDTs). In this work, we revisit the boosting paradigm, traditionally associated with tree ensembles, and ask whether it can be applied as a general training principle for LLM fine-tuning. We propose BoostLLM, a framework that transforms parameter-efficient fine-tuning into a multi-round residual optimization process by training sequential PEFT adapters as weak learners. To incorporate tabular inductive bias, BoostLLM integrates decision-tree paths as a second input view alongside raw features; analysis reveals that the path view acts as a structured teacher in early training steps before the model shifts toward feature-driven representations. Empirically, BoostLLM achieves consistent improvements over standard fine-tuning across multiple LLM backbones and datasets, matching or surpassing XGBoost across a wide range of shot counts and outperforming GPT-4o-based methods with a 4B model. We further show that the framework scales: pairing with stronger tree models and extended boosting horizons yields additional gains under appropriate stabilization. These results suggest that boosting can serve as a general training principle for LLM fine-tuning, particularly in low-data regimes for structured data.