Easy2Hard-Bench: Standardized Difficulty Labels for Profiling LLM Performance and Generalization
This provides a standardized benchmark for evaluating LLM generalization across tasks of varying complexity, which is incremental as it builds on existing difficulty estimation methods.
The authors tackled the lack of datasets with fine-grained difficulty annotations for profiling LLM generalization by introducing Easy2Hard-Bench, a collection of 6 benchmark datasets with numerical difficulty scores derived from human and LLM performance data, and they analyzed six state-of-the-art LLMs to assess their performance across difficulty levels.
While generalization over tasks from easy to hard is crucial to profile language models (LLMs), the datasets with fine-grained difficulty annotations for each problem across a broad range of complexity are still blank. Aiming to address this limitation, we present Easy2Hard-Bench, a consistently formatted collection of 6 benchmark datasets spanning various domains, such as mathematics and programming problems, chess puzzles, and reasoning questions. Each problem within these datasets is annotated with numerical difficulty scores. To systematically estimate problem difficulties, we collect abundant performance data on attempts to each problem by humans in the real world or LLMs on the prominent leaderboard. Leveraging the rich performance data, we apply well-established difficulty ranking systems, such as Item Response Theory (IRT) and Glicko-2 models, to uniformly assign numerical difficulty scores to problems. Moreover, datasets in Easy2Hard-Bench distinguish themselves from previous collections by a higher proportion of challenging problems. Through extensive experiments with six state-of-the-art LLMs, we provide a comprehensive analysis of their performance and generalization capabilities across varying levels of difficulty, with the aim of inspiring future research in LLM generalization. The datasets are available at https://huggingface.co/datasets/furonghuang-lab/Easy2Hard-Bench.