Reasoning Bias of Next Token Prediction Training
This addresses the challenge of optimizing training strategies for reasoning in LLMs, offering insights for researchers and practitioners, though it is incremental as it builds on existing paradigms like CTP.
The paper tackles the problem of training large language models for reasoning by comparing next token prediction (NTP) with critical token prediction (CTP), finding that NTP, despite exposure to noise, surpasses CTP in reasoning ability due to regularization effects, with empirical results showing enhanced generalization and robustness across benchmarks.
Since the inception of Large Language Models (LLMs), the quest to efficiently train them for superior reasoning capabilities has been a pivotal challenge. The dominant training paradigm for LLMs is based on next token prediction (NTP). Alternative methodologies, called Critical Token Prediction (CTP), focused exclusively on specific critical tokens (such as the answer in Q\&A dataset), aiming to reduce the overfitting of extraneous information and noise. Contrary to initial assumptions, our research reveals that despite NTP's exposure to noise during training, it surpasses CTP in reasoning ability. We attribute this counterintuitive outcome to the regularizing influence of noise on the training dynamics. Our empirical analysis shows that NTP-trained models exhibit enhanced generalization and robustness across various benchmark reasoning datasets, demonstrating greater resilience to perturbations and achieving flatter loss minima. These findings illuminate that NTP is instrumental in fostering reasoning abilities during pretraining, whereas CTP is more effective for finetuning, thereby enriching our comprehension of optimal training strategies in LLM development.