Can Large Language Models Understand DL-Lite Ontologies? An Empirical Study
This addresses the problem of evaluating LLMs' capability to understand structured symbolic knowledge for knowledge engineering, but it is incremental as it focuses on a specific ontology type.
The paper empirically analyzed whether large language models (LLMs) can understand DL-Lite ontologies, finding they can handle formal syntax and semantics but struggle with TBox NI transitivity and large ABoxes.
Large language models (LLMs) have shown significant achievements in solving a wide range of tasks. Recently, LLMs' capability to store, retrieve and infer with symbolic knowledge has drawn a great deal of attention, showing their potential to understand structured information. However, it is not yet known whether LLMs can understand Description Logic (DL) ontologies. In this work, we empirically analyze the LLMs' capability of understanding DL-Lite ontologies covering 6 representative tasks from syntactic and semantic aspects. With extensive experiments, we demonstrate both the effectiveness and limitations of LLMs in understanding DL-Lite ontologies. We find that LLMs can understand formal syntax and model-theoretic semantics of concepts and roles. However, LLMs struggle with understanding TBox NI transitivity and handling ontologies with large ABoxes. We hope that our experiments and analyses provide more insights into LLMs and inspire to build more faithful knowledge engineering solutions.