Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models
This addresses the problem of data scarcity and diversity for researchers and practitioners in AI, offering a method to reduce reliance on human-generated data, though it is incremental as it builds on existing self-training techniques.
The paper tackles the limitation of fine-tuning language models on human data by proposing a self-training method called ReST$^{EM}$ that uses scalar feedback, such as correctness verification on math problems, to generate and filter samples for iterative fine-tuning. It shows that this approach scales with model size and significantly outperforms human-data-only fine-tuning on MATH reasoning and APPS coding benchmarks.
Fine-tuning language models~(LMs) on human-generated data remains a prevalent practice. However, the performance of such models is often limited by the quantity and diversity of high-quality human data. In this paper, we explore whether we can go beyond human data on tasks where we have access to scalar feedback, for example, on math problems where one can verify correctness. To do so, we investigate a simple self-training method based on expectation-maximization, which we call ReST$^{EM}$, where we (1) generate samples from the model and filter them using binary feedback, (2) fine-tune the model on these samples, and (3) repeat this process a few times. Testing on advanced MATH reasoning and APPS coding benchmarks using PaLM-2 models, we find that ReST$^{EM}$ scales favorably with model size and significantly surpasses fine-tuning only on human data. Overall, our findings suggest self-training with feedback can substantially reduce dependence on human-generated data.