Scientific and Creative Analogies in Pretrained Language Models
This addresses the challenge of improving analogy understanding in AI for applications requiring creative and scientific reasoning, though it is incremental as it builds on existing datasets and models.
The paper tackled the problem of analogical reasoning in pretrained language models by introducing the SCAN dataset, which tests complex analogies across dissimilar domains, and found that state-of-the-art models perform poorly on these tasks.
This paper examines the encoding of analogy in large-scale pretrained language models, such as BERT and GPT-2. Existing analogy datasets typically focus on a limited set of analogical relations, with a high similarity of the two domains between which the analogy holds. As a more realistic setup, we introduce the Scientific and Creative Analogy dataset (SCAN), a novel analogy dataset containing systematic mappings of multiple attributes and relational structures across dissimilar domains. Using this dataset, we test the analogical reasoning capabilities of several widely-used pretrained language models (LMs). We find that state-of-the-art LMs achieve low performance on these complex analogy tasks, highlighting the challenges still posed by analogy understanding.