EnigmaEval: A Benchmark of Long Multimodal Reasoning Challenges
This provides a novel testbed for evaluating frontier language models on multimodal reasoning, though it is incremental as it builds on existing benchmark efforts.
The authors tackled the need for new challenges to evaluate advanced reasoning in language models by introducing EnigmaEval, a benchmark of 1184 complex multimodal puzzles derived from competitions, where state-of-the-art models achieve extremely low accuracy, revealing shortcomings in unstructured and lateral reasoning.
As language models master existing reasoning benchmarks, we need new challenges to evaluate their cognitive frontiers. Puzzle-solving events are rich repositories of challenging multimodal problems that test a wide range of advanced reasoning and knowledge capabilities, making them a unique testbed for evaluating frontier language models. We introduce EnigmaEval, a dataset of problems and solutions derived from puzzle competitions and events that probes models' ability to perform implicit knowledge synthesis and multi-step deductive reasoning. Unlike existing reasoning and knowledge benchmarks, puzzle solving challenges models to discover hidden connections between seemingly unrelated pieces of information to uncover solution paths. The benchmark comprises 1184 puzzles of varying complexity -- each typically requiring teams of skilled solvers hours to days to complete -- with unambiguous, verifiable solutions that enable efficient evaluation. State-of-the-art language models achieve extremely low accuracy on these puzzles, even lower than other difficult benchmarks such as Humanity's Last Exam, unveiling models' shortcomings when challenged with problems requiring unstructured and lateral reasoning.