Large Language Models Assisting Ontology Evaluation
This work addresses the labor-intensive task of ontology evaluation for ontology engineering experts, but it is incremental as it applies existing LLM methods to a new domain-specific problem.
The paper tackles the problem of costly and error-prone ontology evaluation by introducing OE-Assist, a framework using large language models (LLMs) for automated and semi-automated competency question verification, finding that automated LLM-based evaluation with o1-preview and o3-mini performs similarly to the average user's performance.
Ontology evaluation through functional requirements, such as testing via competency question (CQ) verification, is a well-established yet costly, labour-intensive, and error-prone endeavour, even for ontology engineering experts. In this work, we introduce OE-Assist, a novel framework designed to assist ontology evaluation through automated and semi-automated CQ verification. By presenting and leveraging a dataset of 1,393 CQs paired with corresponding ontologies and ontology stories, our contributions present, to our knowledge, the first systematic investigation into large language model (LLM)-assisted ontology evaluation, and include: (i) evaluating the effectiveness of a LLM-based approach for automatically performing CQ verification against a manually created gold standard, and (ii) developing and assessing an LLM-powered framework to assist CQ verification with Protégé, by providing suggestions. We found that automated LLM-based evaluation with o1-preview and o3-mini perform at a similar level to the average user's performance.