All Claims Are Equal, but Some Claims Are More Equal Than Others: Importance-Sensitive Factuality Evaluation of LLM Generations
This addresses the need for more accurate and robust assessment of LLM factuality, particularly for applications where key information errors are critical, though it is incremental as it builds on existing evaluation frameworks.
The paper tackles the problem that existing factuality evaluation methods for LLM responses treat all claims as equally important, which can be misleading when key information is missing or incorrect. It introduces VITAL, a set of metrics that incorporate claim relevance and importance, and demonstrates on a benchmark of 6,733 queries that VITAL more reliably detects errors in key information than previous methods.
Existing methods for evaluating the factuality of large language model (LLM) responses treat all claims as equally important. This results in misleading evaluations when vital information is missing or incorrect as it receives the same weight as peripheral details, raising the question: how can we reliably detect such differences when there are errors in key information? Current approaches that measure factuality tend to be insensitive to omitted or false key information. To investigate this lack of sensitivity, we construct VITALERRORS, a benchmark of 6,733 queries with minimally altered LLM responses designed to omit or falsify key information. Using this dataset, we demonstrate the insensitivities of existing evaluation metrics to key information errors. To address this gap, we introduce VITAL, a set of metrics that provide greater sensitivity in measuring the factuality of responses by incorporating the relevance and importance of claims with respect to the query. Our analysis demonstrates that VITAL metrics more reliably detect errors in key information than previous methods. Our dataset, metrics, and analysis provide a foundation for more accurate and robust assessment of LLM factuality.