Piecing Together Clues: A Benchmark for Evaluating the Detective Skills of Large Language Models
This work addresses the need for better benchmarks to assess LLMs' detective skills in reading comprehension, though it is incremental as it builds on existing evaluation methods.
The authors tackled the problem of evaluating large language models' ability to detect key information and perform multi-hop reasoning in complex contexts, resulting in the creation of DetectBench, a dataset with 3,928 questions, and the Detective Thinking Framework, which improved model performance.
Detectives frequently engage in information detection and reasoning simultaneously when making decisions across various cases, especially when confronted with a vast amount of information. With the rapid development of large language models~(LLMs), evaluating how these models identify key information and reason to solve questions becomes increasingly relevant. We introduces the DetectBench, a reading comprehension dataset designed to assess a model's ability to jointly ability in key information detection and multi-hop reasoning when facing complex and implicit information. The DetectBench comprises 3,928 questions, each paired with a paragraph averaging 190 tokens in length. To enhance model's detective skills, we propose the Detective Thinking Framework. These methods encourage models to identify all possible clues within the context before reasoning. Our experiments reveal that existing models perform poorly in both information detection and multi-hop reasoning. However, the Detective Thinking Framework approach alleviates this issue.