Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems
This work addresses the issue of reasoning errors in language models for tasks like grade-school math, offering an incremental improvement in accuracy through modified pretraining data.
The paper tackled the problem of language models making reasoning mistakes by incorporating error-correction data into pretraining, showing that this approach improves reasoning accuracy directly without multi-round prompting compared to using error-free data.
Language models have demonstrated remarkable performance in solving reasoning tasks; however, even the strongest models still occasionally make reasoning mistakes. Recently, there has been active research aimed at improving reasoning accuracy, particularly by using pretrained language models to "self-correct" their mistakes via multi-round prompting. In this paper, we follow this line of work but focus on understanding the usefulness of incorporating "error-correction" data directly into the pretraining stage. This data consists of erroneous solution steps immediately followed by their corrections. Using a synthetic math dataset, we show promising results: this type of pretrain data can help language models achieve higher reasoning accuracy directly (i.e., through simple auto-regression, without multi-round prompting) compared to pretraining on the same amount of error-free data. We also delve into many details, such as (1) how this approach differs from beam search, (2) how such data can be prepared, (3) whether masking is needed on the erroneous tokens, (4) the amount of error required, (5) whether such data can be deferred to the fine-tuning stage, and many others.