FinanceQA: A Benchmark for Evaluating Financial Analysis Capabilities of Large Language Models
This addresses the problem of inadequate testing for LLMs in financial analysis, which is crucial for institutions like hedge funds and investment banks, but it is incremental as it builds on existing benchmarking efforts.
The authors introduced FinanceQA, a benchmark to evaluate large language models (LLMs) on complex numerical financial analysis tasks, revealing that current LLMs fail about 60% of realistic tasks, highlighting a gap in capabilities for professional financial work.
FinanceQA is a testing suite that evaluates LLMs' performance on complex numerical financial analysis tasks that mirror real-world investment work. Despite recent advances, current LLMs fail to meet the strict accuracy requirements of financial institutions, with models failing approximately 60% of realistic tasks that mimic on-the-job analyses at hedge funds, private equity firms, investment banks, and other financial institutions. The primary challenges include hand-spreading metrics, adhering to standard accounting and corporate valuation conventions, and performing analysis under incomplete information - particularly in multi-step tasks requiring assumption generation. This performance gap highlights the disconnect between existing LLM capabilities and the demands of professional financial analysis that are inadequately tested by current testing architectures. Results show that higher-quality training data is needed to support such tasks, which we experiment with using OpenAI's fine-tuning API. FinanceQA is publicly released at [this https URL](https://huggingface.co/datasets/AfterQuery/FinanceQA).