Retrieval Or Holistic Understanding? Dolce: Differentiate Our Long Context Evaluation Tasks
This work addresses the need for better evaluation of long-context LLMs by providing a method to differentiate task types, which is incremental as it builds on existing benchmarks without introducing new models.
The paper tackles the problem of distinguishing between retrieval and holistic understanding capabilities in long-context language models by introducing the Dolce framework, which automatically categorizes and measures the difficulty of tasks across 44 benchmarks, finding that 0-67% are retrieval-focused and 0-90% are holistic understanding-focused.
We argue that there are two major distinct capabilities in long context understanding: retrieval and holistic understanding. Understanding and further improving LLMs' long context capabilities would not be possible without knowing the tasks' focus categories. We aim to automatically identify retrieval focused and holistic understanding focused problems from suites of benchmarks and quantitatively measure the difficulty within each focus. In this paper, we present the Dolce framework, which parameterizes each problem by $λ$ (complexity) and $k$ (redundancy) and assigns to one of five predefined focus categories. We propose to sample short contexts from the full context and estimate the probability an LLM solves the problem using the sampled spans. To find the $λ$ and $k$ for each problem, we further propose a mixture model of a non-parametric background noise component and a parametric/non-parametric hybrid oracle component, where we derive the probability functions parameterized by $λ$ and $k$ for both the correct-or-wrong (COW) scenario and the partial-point-in-grading (PIG) scenario. Our proposed methods can identify 0% to 67% of the problems are retrieval focused and 0% to 90% of the problems are holistic understanding focused across 44 existing long context evaluation tasks.