OnionEval: An Unified Evaluation of Fact-conflicting Hallucination for Small-Large Language Models
This work addresses the lack of specific benchmarks for hallucination in small language models, which is an incremental improvement for researchers and practitioners using SLLMs.
The paper tackles the problem of evaluating fact-conflicting hallucinations in small language models (SLLMs) by introducing OnionEval, a multi-layer framework with a context-influence score, which reveals that SLLMs excel in factual analysis but struggle with context reasoning, and shows that a Chain-of-Thought strategy can significantly reduce these limitations.
Large Language Models (LLMs) are highly capable but require significant computational resources for both training and inference. Within the LLM family, smaller models (those with fewer than 10 billion parameters) also perform well across various tasks. However, these smaller models share similar limitations to their larger counterparts, including the tendency to hallucinate. Despite the existence of many benchmarks to evaluate hallucination in LLMs, few have specifically focused on small LLMs (SLLMs). Additionally, SLLMs show widely varying performance across different benchmarks. In this paper, we introduce OnionEval, a multi-layer structured framework with a specific metric called the context-influence score (CI), designed to effectively assess the fact-conflicting hallucination tendencies of small LLMs across different contextual levels. Our experimental results reveal a key feature of SLLMs: they excel in factual analysis but face challenges with context reasoning. Further investigation shows that a simple Chain-of-Thought strategy can significantly reduce these limitations, improving the practical usefulness of SLLMs in real-world applications.