Scalable Oversight for Superhuman AI via Recursive Self-Critiquing
This addresses the problem of ensuring reliable AI alignment for tasks beyond human cognitive limits, though it appears incremental as it builds on existing critique and verification concepts.
The paper tackles the challenge of aligning superhuman AI when human oversight becomes infeasible by proposing recursive self-critiquing, where higher-order critiques are easier than direct evaluation, and finds it a promising approach for scalable oversight.
As AI capabilities increasingly surpass human proficiency in complex tasks, current alignment techniques including SFT and RLHF face fundamental challenges in ensuring reliable oversight. These methods rely on direct human assessment and become untenable when AI outputs exceed human cognitive thresholds. In response to this challenge, we explore two hypotheses: (1) \textit{Critique of critique can be easier than critique itself}, extending the widely-accepted observation that verification is easier than generation to the critique domain, as critique itself is a specialized form of generation; (2) \textit{This difficulty relationship is recursively held}, suggesting that when direct evaluation is infeasible, performing high-order critiques (e.g., critique of critique of critique) offers a more tractable supervision pathway. We further conduct Human-AI and AI-AI experiments to investigate the potential of utilizing recursive self-critiquing for AI supervision. Our results highlight recursive critique as a promising approach for scalable AI oversight.