Compound-QA: A Benchmark for Evaluating LLMs on Compound Questions
This work addresses a gap in LLM evaluation for real-world complex interactions, though it is incremental as it builds on existing QA datasets and methods.
The authors tackled the problem of evaluating large language models (LLMs) on compound questions, which involve multiple sub-questions, by introducing the Compound-QA benchmark derived from existing datasets and showing that LLMs perform significantly worse on these compared to non-compound questions, with methods to improve performance yielding significant gains.
Large language models (LLMs) demonstrate remarkable performance across various tasks, prompting researchers to develop diverse evaluation benchmarks. However, existing benchmarks typically measure the ability of LLMs to respond to individual questions, neglecting the complex interactions in real-world applications. In this paper, we introduce Compound Question Synthesis (CQ-Syn) to create the Compound-QA benchmark, focusing on compound questions with multiple sub-questions. This benchmark is derived from existing QA datasets, annotated with proprietary LLMs and verified by humans for accuracy. It encompasses five categories: Factual-Statement, Cause-and-Effect, Hypothetical-Analysis, Comparison-and-Selection, and Evaluation-and-Suggestion. It evaluates the LLM capability in terms of three dimensions including understanding, reasoning, and knowledge. Our assessment of eight open-source LLMs using Compound-QA reveals distinct patterns in their responses to compound questions, which are significantly poorer than those to non-compound questions. Additionally, we investigate various methods to enhance LLMs performance on compound questions. The results indicate that these approaches significantly improve the models' comprehension and reasoning abilities on compound questions.