FIRE: A Comprehensive Benchmark for Financial Intelligence and Reasoning Evaluation
This addresses the need for standardized evaluation of LLMs in financial applications, though it is incremental as it builds on existing benchmarking practices.
The authors introduced FIRE, a benchmark for evaluating LLMs' theoretical financial knowledge and practical business scenario handling, and tested state-of-the-art models including XuanYuan 4.0, revealing systematic capability boundaries.
We introduce FIRE, a comprehensive benchmark designed to evaluate both the theoretical financial knowledge of LLMs and their ability to handle practical business scenarios. For theoretical assessment, we curate a diverse set of examination questions drawn from widely recognized financial qualification exams, enabling evaluation of LLMs deep understanding and application of financial knowledge. In addition, to assess the practical value of LLMs in real-world financial tasks, we propose a systematic evaluation matrix that categorizes complex financial domains and ensures coverage of essential subdomains and business activities. Based on this evaluation matrix, we collect 3,000 financial scenario questions, consisting of closed-form decision questions with reference answers and open-ended questions evaluated by predefined rubrics. We conduct comprehensive evaluations of state-of-the-art LLMs on the FIRE benchmark, including XuanYuan 4.0, our latest financial-domain model, as a strong in-domain baseline. These results enable a systematic analysis of the capability boundaries of current LLMs in financial applications. We publicly release the benchmark questions and evaluation code to facilitate future research.