StoryAnalogy: Deriving Story-level Analogies from Large Language Models to Unlock Analogical Understanding
This work addresses the challenge of analogical reasoning in AI for narrative understanding, though it is incremental as it focuses on evaluation and dataset creation rather than a breakthrough method.
The paper tackles the problem of evaluating story-level analogy identification and generation by creating a large-scale corpus, StoryAnalogy, with 24K story pairs and human annotations. It finds that tasks are difficult for models like ChatGPT, which achieved only 30% accuracy compared to over 85% for humans, and shows that fine-tuning on this data improves analogy generation to match zero-shot ChatGPT performance.
Analogy-making between narratives is crucial for human reasoning. In this paper, we evaluate the ability to identify and generate analogies by constructing a first-of-its-kind large-scale story-level analogy corpus, \textsc{StoryAnalogy}, which contains 24K story pairs from diverse domains with human annotations on two similarities from the extended Structure-Mapping Theory. We design a set of tests on \textsc{StoryAnalogy}, presenting the first evaluation of story-level analogy identification and generation. Interestingly, we find that the analogy identification tasks are incredibly difficult not only for sentence embedding models but also for the recent large language models (LLMs) such as ChatGPT and LLaMa. ChatGPT, for example, only achieved around 30% accuracy in multiple-choice questions (compared to over 85% accuracy for humans). Furthermore, we observe that the data in \textsc{StoryAnalogy} can improve the quality of analogy generation in LLMs, where a fine-tuned FlanT5-xxl model achieves comparable performance to zero-shot ChatGPT.