Contrastive Weak-to-strong Generalization
This addresses a bottleneck in scaling LLMs without human feedback, offering a more robust method for capability transfer, though it appears incremental as it builds on existing weak-to-strong and contrastive decoding techniques.
The paper tackles the problem of noise and biases in weak-to-strong generalization for scaling large language models, proposing Contrastive Weak-to-Strong Generalization (ConG) to improve sample quality and robustness, with empirical results showing consistent improvements across model families.
Weak-to-strong generalization provides a promising paradigm for scaling large language models (LLMs) by training stronger models on samples from aligned weaker ones, without requiring human feedback or explicit reward modeling. However, its robustness and generalization are hindered by the noise and biases in weak-model outputs, which limit its applicability in practice. To address this challenge, we leverage implicit rewards, which approximate explicit rewards through log-likelihood ratios, and reveal their structural equivalence with Contrastive Decoding (CD), a decoding strategy shown to reduce noise in LLM generation. Building on this connection, we propose Contrastive Weak-to-Strong Generalization (ConG), a framework that employs contrastive decoding between pre- and post-alignment weak models to generate higher-quality samples. This approach enables more reliable capability transfer, denoising, and improved robustness, substantially mitigating the limitations of traditional weak-to-strong methods. Empirical results across different model families confirm consistent improvements, demonstrating the generality and effectiveness of ConG. Taken together, our findings highlight the potential of ConG to advance weak-to-strong generalization and provide a promising pathway toward AGI.