GraphARC: A Comprehensive Benchmark for Graph-Based Abstract Reasoning
This benchmark addresses the problem of evaluating abstract reasoning on graph-structured data for researchers developing graph foundation models, providing a systematic way to assess generalization abilities.
This paper introduces GraphARC, a new benchmark for abstract reasoning on graph-structured data, which generalizes the few-shot transformation learning paradigm of the Abstraction and Reasoning Corpus (ARC). It reveals that state-of-the-art language models struggle with full graph transformation tasks and larger instances, indicating a comprehension-execution gap and scaling barriers.
Relational reasoning lies at the heart of intelligence, but existing benchmarks are typically confined to formats such as grids or text. We introduce GraphARC, a benchmark for abstract reasoning on graph-structured data. GraphARC generalizes the few-shot transformation learning paradigm of the Abstraction and Reasoning Corpus (ARC). Each task requires inferring a transformation rule from a few input-output pairs and applying it to a new test graph, covering local, global, and hierarchical graph transformations. Unlike grid-based ARC, GraphARC instances can be generated at scale across diverse graph families and sizes, enabling systematic evaluation of generalization abilities. We evaluate state-of-the-art language models on GraphARC and observe clear limitations. Models can answer questions about graph properties but often fail to solve the full graph transformation task, revealing a comprehension-execution gap. Performance further degrades on larger instances, exposing scaling barriers. More broadly, by combining aspects of node classification, link prediction, and graph generation within a single framework, GraphARC provides a promising testbed for future graph foundation models.