AnaloBench: Benchmarking the Identification of Abstract and Long-context Analogies
This addresses the challenge of assessing AI's ability to mimic human-like analogical thinking, which is incremental as it builds on prior benchmarking efforts.
The paper tackles the problem of evaluating analogical reasoning in language models by introducing AnaloBench, a benchmark focusing on abstract and long-context analogies, and finds that scaling models offers minimal gains for lengthy scenarios or recalling relevant information from large pools.
Humans regularly engage in analogical thinking, relating personal experiences to current situations (X is analogous to Y because of Z). Analogical thinking allows humans to solve problems in creative ways, grasp difficult concepts, and articulate ideas more effectively. Can language models (LMs) do the same? To answer this question, we propose AnaloBench, a benchmark to determine analogical reasoning ability in LMs. Our benchmarking approach focuses on aspects of this ability that are common among humans: (i) recalling related experiences from a large amount of information, and (ii) applying analogical reasoning to complex and lengthy scenarios. We test a broad collection of proprietary models (e.g., GPT family, Claude V2) and open source models such as LLaMA2. As in prior results, scaling up LMs results in some performance boosts. Surprisingly, scale offers minimal gains when, (i) analogies involve lengthy scenarios, or (ii) recalling relevant scenarios from a large pool of information, a process analogous to finding a needle in a haystack. We hope these observations encourage further research in this field.