Continuous Self-Improvement of Large Language Models by Test-time Training with Verifier-Driven Sample Selection
This addresses the problem of LLMs faltering on novel reasoning tasks for AI practitioners, offering an incremental efficiency gain in adaptation.
The paper tackles the challenge of adapting pretrained language models to unlabeled, out-of-distribution data by introducing VDS-TTT, a framework that uses a verifier to select high-confidence pseudo-labeled examples for test-time training, resulting in up to a 32.29% relative improvement over the base model.
Learning to adapt pretrained language models to unlabeled, out-of-distribution data is a critical challenge, as models often falter on structurally novel reasoning tasks even while excelling within their training distribution. We introduce a new framework called VDS-TTT - Verifier-Driven Sample Selection for Test-Time Training to efficiently address this. We use a learned verifier to score a pool of generated responses and select only from high ranking pseudo-labeled examples for fine-tuned adaptation. Specifically, for each input query our LLM generates N candidate answers; the verifier assigns a reliability score to each, and the response with the highest confidence and above a fixed threshold is paired with its query for test-time training. We fine-tune only low-rank LoRA adapter parameters, ensuring adaptation efficiency and fast convergence. Our proposed self-supervised framework is the first to synthesize verifier driven test-time training data for continuous self-improvement of the model. Experiments across three diverse benchmarks and three state-of-the-art LLMs demonstrate that VDS-TTT yields up to a 32.29% relative improvement over the base model and a 6.66% gain compared to verifier-based methods without test-time training, highlighting its effectiveness and efficiency for on-the-fly large language model adaptation.