Learning to Reason via Self-Iterative Process Feedback for Small Language Models
This addresses the challenge of enhancing reasoning in SLMs without costly external signals, though it appears incremental as it builds on existing preference optimization techniques.
This paper tackles the problem of small language models (SLMs) underperforming in reasoning tasks by enabling them to learn from self-iterative feedback, improving Gemma-2B's accuracy by 12.43 on GSM8K and 3.95 on MBPP compared to supervised fine-tuning.
Small language models (SLMs) are more efficient, cost-effective, and customizable than large language models (LLMs), though they often underperform in specific areas like reasoning. Past methods for enhancing SLMs' reasoning, such as supervised fine-tuning and distillation, often depend on costly external signals, resulting in SLMs being overly confident with limited supervision signals, thus limiting their abilities. Therefore, this study enables SLMs to learn to reason from self-iterative feedback. By combining odds ratio preference optimization (ORPO), we fine-tune and align SLMs using positive and negative signals generated by themselves. Additionally, we introduce process supervision for rewards in preference alignment by sampling-based inference simulation and process reward models. Compared to Supervised Fine-Tuning (SFT), our method improves the performance of Gemma-2B by 12.43 (Acc) on GSM8K and 3.95 (Pass@1) on MBPP. Furthermore, the proposed method also demonstrated superior out-of-domain generalization capabilities on MMLU_Math and HumanEval.