A Comparative Study of Competency Question Elicitation Methods from Ontology Requirements
This study addresses the need for systematic evaluation of CQ elicitation methods in knowledge engineering, though it is incremental as it builds on existing approaches without introducing a new paradigm.
The paper tackled the problem of comparing different methods for eliciting Competency Questions (CQs) from ontology requirements, finding that manual formulation, pattern instantiation, and LLM generation produce CQs with varying characteristics, and LLMs can serve as an initial step but require refinement.
Competency Questions (CQs) are pivotal in knowledge engineering, guiding the design, validation, and testing of ontologies. A number of diverse formulation approaches have been proposed in the literature, ranging from completely manual to Large Language Model (LLM) driven ones. However, attempts to characterise the outputs of these approaches and their systematic comparison are scarce. This paper presents an empirical comparative evaluation of three distinct CQ formulation approaches: manual formulation by ontology engineers, instantiation of CQ patterns, and generation using state of the art LLMs. We generate CQs using each approach from a set of requirements for cultural heritage, and assess them across different dimensions: degree of acceptability, ambiguity, relevance, readability and complexity. Our contribution is twofold: (i) the first multi-annotator dataset of CQs generated from the same source using different methods; and (ii) a systematic comparison of the characteristics of the CQs resulting from each approach. Our study shows that different CQ generation approaches have different characteristics and that LLMs can be used as a way to initially elicit CQs, however these are sensitive to the model used to generate CQs and they generally require a further refinement step before they can be used to model requirements.