Marathon: A Race Through the Realm of Long Context with Large Language Models
This addresses the need for better evaluation tools for researchers and developers working with long-context LLMs, though it is incremental as it builds on existing benchmark concepts.
The authors tackled the problem of evaluating large language models' comprehension and reasoning in long contexts by introducing Marathon, a multiple-choice benchmark that overcomes limitations of existing metrics, and they conducted evaluations showing it provides a rapid, precise, and unbiased assessment.
With the advancement of large language models (LLMs) and the expansion of their context windows, existing long-context benchmarks fall short in effectively evaluating the models' comprehension and reasoning abilities in extended texts. Moreover, conventional benchmarks relying on F1 metrics often inaccurately score responses: they may undervalue correct answers that differ from the reference responses and overvalue incorrect ones that resemble the reference texts. In response to these limitations, we introduce Marathon, a novel evaluation benchmark that adopts a multiple-choice question format. It is specifically designed to overcome the constraints of previous benchmarks and provide a rapid, precise, and unbiased appraisal of the long-context comprehension skills of large language models. We conducted comprehensive evaluations on the Marathon benchmark with a range of state-of-the-art LLMs and assessed the effectiveness of various optimization strategies tailored for long-context generation. We anticipate that the Marathon benchmark and its associated leaderboard will enable a more precise and equitable evaluation of LLMs' capabilities in understanding and reasoning over extended contexts. Marathon is available at https://github.com/Hambaobao/Marathon.