Weak-to-Strong Reasoning
This work addresses the challenge of scalable supervision for advanced AI reasoning, offering a method to improve model performance without relying on stronger models or human annotations, though it appears incremental as it builds on weak-to-strong learning concepts.
The paper tackles the problem of supervising large language models (LLMs) when they exceed human-level capabilities by introducing a progressive learning framework that enables strong models to autonomously refine training data without advanced models or human input, resulting in significant enhancements in reasoning capabilities on datasets like GSM8K and MATH, validated with models such as Llama2-70b and Llama3-70b.
When large language models (LLMs) exceed human-level capabilities, it becomes increasingly challenging to provide full-scale and accurate supervision for these models. Weak-to-strong learning, which leverages a less capable model to unlock the latent abilities of a stronger model, proves valuable in this context. Yet, the efficacy of this approach for complex reasoning tasks is still untested. Furthermore, tackling reasoning tasks under the weak-to-strong setting currently lacks efficient methods to avoid blindly imitating the weak supervisor including its errors. In this paper, we introduce a progressive learning framework that enables the strong model to autonomously refine its training data, without requiring input from either a more advanced model or human-annotated data. This framework begins with supervised fine-tuning on a selective small but high-quality dataset, followed by preference optimization on contrastive samples identified by the strong model itself. Extensive experiments on the GSM8K and MATH datasets demonstrate that our method significantly enhances the reasoning capabilities of Llama2-70b using three separate weak models. This method is further validated in a forward-looking experimental setup, where Llama3-8b-instruct effectively supervises Llama3-70b on the highly challenging OlympicArena dataset. This work paves the way for a more scalable and sophisticated strategy to enhance AI reasoning powers. All relevant code and resources are available in \url{https://github.com/GAIR-NLP/weak-to-strong-reasoning}.