COLUMBUS: Evaluating COgnitive Lateral Understanding through Multiple-choice reBUSes
This addresses the problem of evaluating cognitive lateral understanding in AI systems, which is incremental as it introduces a new benchmark for an understudied area.
The authors tackled the lack of benchmarks for lateral thinking in AI by creating COLUMBUS, a synthetic benchmark with over 1,000 multiple-choice rebus puzzles, and found that state-of-the-art vision-language models perform decently but show a substantial gap compared to humans.
While visual question-answering (VQA) benchmarks have catalyzed the development of reasoning techniques, they have focused on vertical thinking. Effective problem-solving also necessitates lateral thinking, which remains understudied in AI and has not been used to test visual perception systems. To bridge this gap, we formulate visual lateral thinking as a multiple-choice question-answering task and describe a three-step taxonomy-driven methodology for instantiating task examples. Then, we develop COLUMBUS, a synthetic benchmark that applies the task pipeline to create QA sets with text and icon rebus puzzles based on publicly available collections of compounds and common phrases. COLUMBUS comprises over 1,000 puzzles, each with four answer candidates. While the SotA vision-language models (VLMs) achieve decent performance, our evaluation demonstrates a substantial gap between humans and models. VLMs benefit from human-curated descriptions but struggle to self-generate such representations at the right level of abstraction.