Weak-to-Strong Generalization under Distribution Shifts
This addresses a critical issue for deploying superhuman AI models by ensuring robust supervision under distribution shifts, representing a novel method for a known bottleneck.
The paper tackles the problem of weak-to-strong generalization failing under distribution shifts, where strong models perform worse than weak supervisors, and proposes RAVEN, a framework that dynamically combines weak models to supervise strong ones, achieving over 30% improvement on out-of-distribution tasks while matching or surpassing in-distribution performance.
As future superhuman models become increasingly complex, accurately supervising their behavior may exceed human capabilities. Recent works have demonstrated that in such scenarios, weak models can effectively supervise strong models, a phenomenon known as weak-to-strong generalization. However, we find that naive weak-to-strong generalization fails under distribution shifts, often leading to worse performance of the strong model than its weak supervisors. To address this, we propose RAVEN, a robust weak-to-strong generalization framework that dynamically learns the optimal combinations of weak models in addition to parameters of the strong model. We demonstrate the effectiveness of RAVEN on image classification, text classification, and preference alignment tasks. RAVEN outperforms alternative baselines by over 30% on out-of-distribution tasks while matching or surpassing existing methods on in-distribution tasks. Moreover, our results show that RAVEN assigns higher weights to more accurate weak models, demonstrating its ability to automatically identify trustworthy supervision.