OAEI-LLM-T: A TBox Benchmark Dataset for Understanding Large Language Model Hallucinations in Ontology Matching
This addresses hallucinations in LLM-based ontology matching systems, providing a tool for evaluation and improvement, but it is incremental as it builds on existing datasets.
The paper tackles the problem of hallucinations in large language models (LLMs) for ontology matching by introducing a new benchmark dataset, OAEI-LLM-T, which captures hallucinations from ten LLMs and organizes them into categories, showing its usefulness for leaderboards and fine-tuning.
Hallucinations are often inevitable in downstream tasks using large language models (LLMs). To tackle the substantial challenge of addressing hallucinations for LLM-based ontology matching (OM) systems, we introduce a new benchmark dataset OAEI-LLM-T. The dataset evolves from seven TBox datasets in the Ontology Alignment Evaluation Initiative (OAEI), capturing hallucinations of ten different LLMs performing OM tasks. These OM-specific hallucinations are organised into two primary categories and six sub-categories. We showcase the usefulness of the dataset in constructing an LLM leaderboard for OM tasks and for fine-tuning LLMs used in OM tasks.