Debatable Intelligence: Benchmarking LLM Judges via Debate Speech Evaluation
This work addresses the need for systematic benchmarking of LLMs in complex cognitive tasks like debate evaluation, though it is incremental as it builds on existing LLM assessment methods.
The paper tackles the problem of assessing LLM judges by introducing Debate Speech Evaluation as a benchmark, revealing that while larger models approximate human judgments in some respects, they differ substantially overall, with frontier LLMs performing at a human level in generating persuasive speeches.
We introduce Debate Speech Evaluation as a novel and challenging benchmark for assessing LLM judges. Evaluating debate speeches requires a deep understanding of the speech at multiple levels, including argument strength and relevance, the coherence and organization of the speech, the appropriateness of its style and tone, and so on. This task involves a unique set of cognitive abilities that previously received limited attention in systematic LLM benchmarking. To explore such skills, we leverage a dataset of over 600 meticulously annotated debate speeches and present the first in-depth analysis of how state-of-the-art LLMs compare to human judges on this task. Our findings reveal a nuanced picture: while larger models can approximate individual human judgments in some respects, they differ substantially in their overall judgment behavior. We also investigate the ability of frontier LLMs to generate persuasive, opinionated speeches, showing that models may perform at a human level on this task.