FinLFQA: Evaluating Attributed Text Generation of LLMs in Financial Long-Form Question Answering
This addresses the need for reliable attribution in financial applications, though it is incremental as it builds on existing attribution benchmarks by adding domain-specific complexity.
The paper tackles the problem of LLMs hallucinating in long-form financial question answering by introducing FinLFQA, a benchmark that evaluates nuanced attribution including evidence extraction, numerical reasoning, and domain knowledge, finding that fine-grained metrics are crucial and end-to-end generation performs comparably to post-hoc methods.
Large Language Models (LLMs) frequently hallucinate to long-form questions, producing plausible yet factually incorrect answers. A common mitigation strategy is to provide attribution to LLM outputs. However, existing benchmarks primarily focus on simple attribution that retrieves supporting textual evidence as references. We argue that in real-world scenarios such as financial applications, attribution goes beyond reference retrieval. We introduce FinLFQA, a benchmark designed to evaluate the ability of LLMs to generate long-form answers to complex financial questions with reliable and nuanced attributions. FinLFQA evaluates three critical aspects of attribution through human annotations: (1) supporting evidence extracted from financial reports, (2) intermediate numerical reasoning steps, and (3) domain-specific financial knowledge that informs the reasoning process. We further provide an automatic evaluation framework covering both answer quality and attribution quality. Through extensive experiments on eight LLMs across multiple attribution-generation paradigms, we find that fine-grained metrics are important to distinguish model capabilities, that end-to-end generation achieves comparable performance to post-hoc approaches, and that iterative refinement only helps when guided by external feedback.