ZOGRASCOPE: A New Benchmark for Semantic Parsing over Property Graphs
This addresses a gap in semantic parsing research for property graphs, which are widely used in industry, but is incremental as it focuses on benchmarking rather than novel methods.
The authors tackled the lack of evaluation resources for semantic parsing over property graphs by introducing ZOGRASCOPE, a benchmark with manually annotated Cypher queries, and tested various LLMs, achieving performance metrics across different settings.
In recent years, the need for natural language interfaces to knowledge graphs has become increasingly important since they enable easy and efficient access to the information contained in them. In particular, property graphs (PGs) have seen increased adoption as a means of representing complex structured information. Despite their growing popularity in industry, PGs remain relatively underrepresented in semantic parsing research with a lack of resources for evaluation. To address this gap, we introduce ZOGRASCOPE, a benchmark designed specifically for PGs and queries written in Cypher. Our benchmark includes a diverse set of manually annotated queries of varying complexity and is organized into three partitions: iid, compositional and length. We complement this paper with a set of experiments that test the performance of different LLMs in a variety of learning settings.