DebateQA: Evaluating Question Answering on Debatable Knowledge
This addresses the need for reliable evaluation of LLMs on inherently debatable questions, which is incremental as it builds on existing QA benchmarks by adding debatable aspects.
The authors tackled the problem of evaluating question answering on debatable knowledge by introducing DebateQA, a dataset of 2,941 debatable questions with human-annotated partial answers, and developed two metrics that align with human preferences and are stable across models, revealing that LLMs vary in providing comprehensive perspectives.
The rise of large language models (LLMs) has enabled us to seek answers to inherently debatable questions on LLM chatbots, necessitating a reliable way to evaluate their ability. However, traditional QA benchmarks assume fixed answers are inadequate for this purpose. To address this, we introduce DebateQA, a dataset of 2,941 debatable questions, each accompanied by multiple human-annotated partial answers that capture a variety of perspectives. We develop two metrics: Perspective Diversity, which evaluates the comprehensiveness of perspectives, and Dispute Awareness, which assesses if the LLM acknowledges the question's debatable nature. Experiments demonstrate that both metrics align with human preferences and are stable across different underlying models. Using DebateQA with two metrics, we assess 12 popular LLMs and retrieval-augmented generation methods. Our findings reveal that while LLMs generally excel at recognizing debatable issues, their ability to provide comprehensive answers encompassing diverse perspectives varies considerably.