Are Query-Based Ontology Debuggers Really Helping Knowledge Engineers?
This work addresses the challenge of improving debugging tools for knowledge engineers in complex systems like biomedical ontologies, though it is incremental as it validates an existing approach rather than introducing a new one.
The paper tackled the problem of evaluating the practical effectiveness of query-based ontology debugging tools for knowledge engineers, finding through user studies that the interactive approach is more efficient than algorithmic debugging but that users often make errors, highlighting the need for careful query design.
Real-world semantic or knowledge-based systems, e.g., in the biomedical domain, can become large and complex. Tool support for the localization and repair of faults within knowledge bases of such systems can therefore be essential for their practical success. Correspondingly, a number of knowledge base debugging approaches, in particular for ontology-based systems, were proposed throughout recent years. Query-based debugging is a comparably recent interactive approach that localizes the true cause of an observed problem by asking knowledge engineers a series of questions. Concrete implementations of this approach exist, such as the OntoDebug plug-in for the ontology editor Protégé. To validate that a newly proposed method is favorable over an existing one, researchers often rely on simulation-based comparisons. Such an evaluation approach however has certain limitations and often cannot fully inform us about a method's true usefulness. We therefore conducted different user studies to assess the practical value of query-based ontology debugging. One main insight from the studies is that the considered interactive approach is indeed more efficient than an alternative algorithmic debugging based on test cases. We also observed that users frequently made errors in the process, which highlights the importance of a careful design of the queries that users need to answer.