The Capabilities and Limitations of Weak-to-Strong Generalization: Generalization and Calibration
This work addresses the challenge of aligning superhuman AI models with human values, offering incremental theoretical insights into a promising but limited approach.
The paper tackles the problem of understanding weak-to-strong generalization by providing theoretical bounds on generalization and calibration errors, revealing that weak model performance and optimization balance are critical, and extends results to regression with KL divergence, showing the strong model can outperform the weak teacher by their disagreement magnitude.
Weak-to-strong generalization, where weakly supervised strong models outperform their weaker teachers, offers a promising approach to aligning superhuman models with human values. To deepen the understanding of this approach, we provide theoretical insights into its capabilities and limitations. First, in the classification setting, we establish upper and lower generalization error bounds for the strong model, identifying the primary limitations as stemming from the weak model's generalization error and the optimization objective itself. Additionally, we derive lower and upper bounds on the calibration error of the strong model. These theoretical bounds reveal two critical insights: (1) the weak model should demonstrate strong generalization performance and maintain well-calibrated predictions, and (2) the strong model's training process must strike a careful balance, as excessive optimization could undermine its generalization capability by over-relying on the weak supervision signals. Finally, in the regression setting, we extend the work of Charikar et al. (2024) to a loss function based on Kullback-Leibler (KL) divergence, offering guarantees that the strong student can outperform its weak teacher by at least the magnitude of their disagreement. We conduct sufficient experiments to validate our theory.