Knowledge Divergence and the Value of Debate for Scalable Oversight
This work provides a formal framework for understanding when adversarial oversight protocols, like debate, are beneficial for scalable AI safety, particularly when integrating knowledge from models with complementary information.
This paper analyzes the value of debate in AI safety by parameterizing it through knowledge divergence between debating models, proving that debate advantage has a closed form. It shows that debate offers significant benefits when models have divergent knowledge, achieving outcomes inaccessible to individual models, but can fail due to adversarial incentives in the compositional regime.
AI safety via debate and reinforcement learning from AI feedback (RLAIF) are both proposed methods for scalable oversight of advanced AI systems, yet no formal framework relates them or characterizes when debate offers an advantage. We analyze this by parameterizing debate's value through the geometry of knowledge divergence between debating models. Using principal angles between models' representation subspaces, we prove that the debate advantage admits an exact closed form. When models share identical training corpora, debate reduces to RLAIF-like where a single-agent method recovers the same optimum. When models possess divergent knowledge, debate advantage scales with a phase transition from quadratic regime (debate offers negligible benefit) to linear regime (debate is essential). We classify three regimes of knowledge divergence (shared, one-sided, and compositional) and provide existence results showing that debate can achieve outcomes inaccessible to either model alone, alongside a negative result showing that sufficiently strong adversarial incentives cause coordination failure in the compositional regime, with a sharp threshold separating effective from ineffective debate. We offer the first formal connection between debate and RLAIF, a geometric foundation for understanding when adversarial oversight protocols are justified, and connection to the problem of eliciting latent knowledge across models with complementary information.