KACC: A Multi-task Benchmark for Knowledge Abstraction, Concretization and Completion
This work addresses the need for thorough analysis of KACC abilities in AI models, which is crucial for simulating human-like knowledge management, but it is incremental as it builds upon and improves existing benchmarks.
The authors tackled the lack of a comprehensive benchmark for evaluating models on knowledge abstraction, concretization, and completion (KACC) by proposing KACC, a unified knowledge graph benchmark with larger datasets, novel tasks like multi-hop knowledge abstraction and concretization, and harder samples, which experimental results show to be challenging for existing methods.
A comprehensive knowledge graph (KG) contains an instance-level entity graph and an ontology-level concept graph. The two-view KG provides a testbed for models to "simulate" human's abilities on knowledge abstraction, concretization, and completion (KACC), which are crucial for human to recognize the world and manage learned knowledge. Existing studies mainly focus on partial aspects of KACC. In order to promote thorough analyses for KACC abilities of models, we propose a unified KG benchmark by improving existing benchmarks in terms of dataset scale, task coverage, and difficulty. Specifically, we collect new datasets that contain larger concept graphs, abundant cross-view links as well as dense entity graphs. Based on the datasets, we propose novel tasks such as multi-hop knowledge abstraction (MKA), multi-hop knowledge concretization (MKC) and then design a comprehensive benchmark. For MKA and MKC tasks, we further annotate multi-hop hierarchical triples as harder samples. The experimental results of existing methods demonstrate the challenges of our benchmark. The resource is available at https://github.com/thunlp/KACC.