Towards Long Context Hallucination Detection
This addresses the open problem of detecting hallucinations in long-context LLM inputs, which is an incremental improvement over existing methods for contextual hallucination detection.
The paper tackles the problem of contextual hallucination in large language models (LLMs) for long-context inputs by constructing a specialized dataset and proposing a novel architecture that enables pre-trained encoder models to process long contexts. The results show that their architecture significantly outperforms previous models of similar size and LLM-based models across various metrics while providing substantially faster inference.
Large Language Models (LLMs) have demonstrated remarkable performance across various tasks. However, they are prone to contextual hallucination, generating information that is either unsubstantiated or contradictory to the given context. Although many studies have investigated contextual hallucinations in LLMs, addressing them in long-context inputs remains an open problem. In this work, we take an initial step toward solving this problem by constructing a dataset specifically designed for long-context hallucination detection. Furthermore, we propose a novel architecture that enables pre-trained encoder models, such as BERT, to process long contexts and effectively detect contextual hallucinations through a decomposition and aggregation mechanism. Our experimental results show that the proposed architecture significantly outperforms previous models of similar size as well as LLM-based models across various metrics, while providing substantially faster inference.