SECQUE: A Benchmark for Evaluating Real-World Financial Analysis Capabilities
This work addresses the need for standardized evaluation of AI models in real-world financial analysis, though it is incremental as it builds on existing benchmarking practices.
The authors introduced SECQUE, a benchmark with 565 expert-written questions for evaluating large language models in financial analysis tasks, and developed SECQUE-Judge, an evaluation mechanism that aligns well with human assessments.
We introduce SECQUE, a comprehensive benchmark for evaluating large language models (LLMs) in financial analysis tasks. SECQUE comprises 565 expert-written questions covering SEC filings analysis across four key categories: comparison analysis, ratio calculation, risk assessment, and financial insight generation. To assess model performance, we develop SECQUE-Judge, an evaluation mechanism leveraging multiple LLM-based judges, which demonstrates strong alignment with human evaluations. Additionally, we provide an extensive analysis of various models' performance on our benchmark. By making SECQUE publicly available, we aim to facilitate further research and advancements in financial AI.