OAEI-LLM: A Benchmark Dataset for Understanding Large Language Model Hallucinations in Ontology Matching
This work addresses the need for benchmarks to understand LLM hallucinations in domain-specific tasks like ontology matching, which is incremental as it builds on prior datasets.
The paper tackles the problem of hallucinations in large language models (LLMs) when applied to ontology matching (OM) by introducing the OAEI-LLM dataset, an extended benchmark based on existing OAEI datasets to evaluate LLM-specific hallucinations in OM tasks.
Hallucinations of large language models (LLMs) commonly occur in domain-specific downstream tasks, with no exception in ontology matching (OM). The prevalence of using LLMs for OM raises the need for benchmarks to better understand LLM hallucinations. The OAEI-LLM dataset is an extended version of the Ontology Alignment Evaluation Initiative (OAEI) datasets that evaluate LLM-specific hallucinations in OM tasks. We outline the methodology used in dataset construction and schema extension, and provide examples of potential use cases.