Learning from Saturated Data: Signals Beyond Correctness for LLM Training
For LLM researchers, this work explores extracting additional training signal from saturated benchmarks, but the results are mixed and task-dependent, indicating an incremental advance.
The paper investigates whether training on questions already solved with perfect accuracy can still improve LLM performance by using fine-grained quality signals (pairwise self-judgments and token-level entropy) instead of binary correctness. On a simple arithmetic task, quality-based training improved performance by up to 18.6% over the base model, but gains on GSM8K were modest and inconsistent.
The growing capabilities of large language models (LLMs) have led to the saturation of many benchmarks and training datasets used to improve them. Motivated by this, we investigate whether questions solved with perfect empirical accuracy can nevertheless be used to improve downstream performance. To do so, we replace binary correctness with two sources of more fine-grained quality signals: (1) pairwise LLM self-judgments, in which the model evaluates the relative quality of its own solutions, and (2) token-level entropy, where token-level uncertainty is used as a proxy for solution quality. We incorporate these signals into several training algorithms and evaluate them on Qwen3-1.7B-Base. When training exclusively on a simple arithmetic task, quality-based signals improve performance by up to $18.6\%$ over the base model, substantially outperforming SFT. On GSM8K, however, gains are more modest and depend strongly on the quality signal. For instance, self-judgments show poor agreement with a stronger external judge and can even degrade performance below the base model. Overall, our results suggest that quality-based training can extract useful signal from saturated questions for base models, but that applying such signals to more complex tasks requires careful calibration and further study.