Benchmarking and Understanding Compositional Relational Reasoning of LLMs
This work addresses the challenge of understanding and benchmarking compositional relational reasoning in LLMs, which is incremental as it builds on mechanistic interpretability studies.
The authors tackled the problem of assessing compositional relational reasoning in large language models by introducing the Generalized Associative Recall benchmark, which revealed fundamental deficiencies in existing models. They identified key attention heads representing abstract true/false notions that are crucial for reasoning across models and tasks.
Compositional relational reasoning (CRR) is a hallmark of human intelligence, but we lack a clear understanding of whether and how existing transformer large language models (LLMs) can solve CRR tasks. To enable systematic exploration of the CRR capability of LLMs, we first propose a new synthetic benchmark called Generalized Associative Recall (GAR) by integrating and generalizing the essence of several tasks in mechanistic interpretability (MI) study in a unified framework. Evaluation shows that GAR is challenging enough for existing LLMs, revealing their fundamental deficiency in CRR. Meanwhile, it is easy enough for systematic MI study. Then, to understand how LLMs solve GAR tasks, we use attribution patching to discover the core circuits reused by Vicuna-33B across different tasks and a set of vital attention heads. Intervention experiments show that the correct functioning of these heads significantly impacts task performance. Especially, we identify two classes of heads whose activations represent the abstract notion of true and false in GAR tasks respectively. They play a fundamental role in CRR across various models and tasks. The dataset and code are available at https://github.com/Caiyun-AI/GAR.