Training Language Models to Win Debates with Self-Play Improves Judge Accuracy
This work addresses the challenge of providing high-quality supervision for difficult-to-evaluate tasks in AI, though it is incremental as it builds on existing debate methods.
The researchers tackled the problem of scalable oversight by training language models to debate via self-play, finding that evaluators answered reading comprehension questions 10% more accurately when judging debate-optimized models compared to consultancy baselines.
We test the robustness of debate as a method of scalable oversight by training models to debate with data generated via self-play. In a long-context reading comprehension task, we find that language model based evaluators answer questions more accurately when judging models optimized to win debates. By contrast, we find no such relationship for consultancy models trained to persuade a judge without an opposing debater present. In quantitative and qualitative comparisons between our debate models and novel consultancy baselines, we find evidence that debate training encourages stronger and more informative arguments, showing promise that it can help provide high-quality supervision for tasks that are difficult to directly evaluate.