Improving Language Model Reasoning with Self-motivated Learning
This addresses the problem of high annotation costs for reasoning data in AI, offering an incremental improvement by automating rationale generation.
The paper tackles the scarcity of high-quality rationales for training language models by introducing a Self-motivated Learning framework that enables models to automatically generate and improve rationales, leading to significant improvements in reasoning ability, such as outperforming text-davinci-002 on some datasets with Llama2 7B.
Large-scale high-quality training data is important for improving the performance of models. After trained with data that has rationales (reasoning steps), models gain reasoning capability. However, the dataset with high-quality rationales is relatively scarce due to the high annotation cost. To address this issue, we propose \textit{Self-motivated Learning} framework. The framework motivates the model itself to automatically generate rationales on existing datasets. Based on the inherent rank from correctness across multiple rationales, the model learns to generate better rationales, leading to higher reasoning capability. Specifically, we train a reward model with the rank to evaluate the quality of rationales, and improve the performance of reasoning through reinforcement learning. Experiment results of Llama2 7B on multiple reasoning datasets show that our method significantly improves the reasoning ability of models, even outperforming text-davinci-002 in some datasets.