Capturing Symmetry and Antisymmetry in Language Models through Symmetry-Aware Training Objectives
This work addresses a gap in relational understanding for language models, which is crucial for applications like knowledge representation, but it is incremental as it builds on existing contrastive learning methods.
The paper tackled the challenge of capturing symmetric and antisymmetric relations in language models by introducing a novel Wikidata-derived dataset, finding that LLMs perform near random chance on it, and addressed this through encoder retraining with contrastive learning, which matched fine-tuned performance while improving efficiency and mitigating forgetting.
Capturing symmetric (e.g., country borders another country) and antisymmetric (e.g., parent_of) relations is crucial for a variety of applications. This paper tackles this challenge by introducing a novel Wikidata-derived natural language inference dataset designed to evaluate large language models (LLMs). Our findings reveal that LLMs perform comparably to random chance on this benchmark, highlighting a gap in relational understanding. To address this, we explore encoder retraining via contrastive learning with k-nearest neighbors. The retrained encoder matches the performance of fine-tuned classification heads while offering additional benefits, including greater efficiency in few-shot learning and improved mitigation of catastrophic forgetting.