FinMR: A Knowledge-Intensive Multimodal Benchmark for Advanced Financial Reasoning
This addresses the problem of evaluating advanced financial reasoning in AI models for researchers and practitioners, though it is incremental as it focuses on creating a new benchmark rather than a novel method.
The authors tackled the lack of rigorous evaluation datasets for multimodal large language models in finance by introducing FinMR, a high-quality, knowledge-intensive multimodal dataset with over 3,200 question-answer pairs, which revealed significant performance gaps between leading models and professional financial analysts.
Multimodal Large Language Models (MLLMs) have made substantial progress in recent years. However, their rigorous evaluation within specialized domains like finance is hindered by the absence of datasets characterized by professional-level knowledge intensity, detailed annotations, and advanced reasoning complexity. To address this critical gap, we introduce FinMR, a high-quality, knowledge-intensive multimodal dataset explicitly designed to evaluate expert-level financial reasoning capabilities at a professional analyst's standard. FinMR comprises over 3,200 meticulously curated and expertly annotated question-answer pairs across 15 diverse financial topics, ensuring broad domain diversity and integrating sophisticated mathematical reasoning, advanced financial knowledge, and nuanced visual interpretation tasks across multiple image types. Through comprehensive benchmarking with leading closed-source and open-source MLLMs, we highlight significant performance disparities between these models and professional financial analysts, uncovering key areas for model advancement, such as precise image analysis, accurate application of complex financial formulas, and deeper contextual financial understanding. By providing richly varied visual content and thorough explanatory annotations, FinMR establishes itself as an essential benchmark tool for assessing and advancing multimodal financial reasoning toward professional analyst-level competence.