FinDeepResearch: Evaluating Deep Research Agents in Rigorous Financial Analysis
This work addresses the problem of evaluating AI agents in complex financial tasks for researchers and developers, though it is incremental as it focuses on benchmarking rather than a new method.
The paper tackles the lack of rigorous evaluation for Deep Research agents in financial analysis by proposing HisRubric, a hierarchical evaluation framework, and the FinDeepResearch benchmark with 15,808 grading items across 64 companies, 8 markets, and 4 languages, revealing strengths and limitations of 16 methods including DR agents and LLMs.
Deep Research (DR) agents, powered by advanced Large Language Models (LLMs), have recently garnered increasing attention for their capability in conducting complex research tasks. However, existing literature lacks a rigorous and systematic evaluation of DR Agent's capabilities in critical research analysis. To address this gap, we first propose HisRubric, a novel evaluation framework with a hierarchical analytical structure and a fine-grained grading rubric for rigorously assessing DR agents' capabilities in corporate financial analysis. This framework mirrors the professional analyst's workflow, progressing from data recognition to metric calculation, and finally to strategic summarization and interpretation. Built on this framework, we construct a FinDeepResearch benchmark that comprises 64 listed companies from 8 financial markets across 4 languages, encompassing a total of 15,808 grading items. We further conduct extensive experiments on the FinDeepResearch using 16 representative methods, including 6 DR agents, 5 LLMs equipped with both deep reasoning and search capabilities, and 5 LLMs with deep reasoning capabilities only. The results reveal the strengths and limitations of these approaches across diverse capabilities, financial markets, and languages, offering valuable insights for future research and development. The benchmark and evaluation code will be made publicly available.