EDINET-Bench: Evaluating LLMs on Complex Financial Tasks using Japanese Financial Statements
This addresses a gap for researchers and practitioners in financial analytics by providing a specialized benchmark, though it is incremental as it focuses on dataset creation rather than novel methods.
The authors tackled the lack of challenging financial datasets for evaluating LLMs in Japanese finance by introducing EDINET-Bench, an open-source benchmark based on 10 years of annual reports, and found that state-of-the-art LLMs performed only slightly better than logistic regression in tasks like fraud detection and earnings forecasting.
Financial analysis presents complex challenges that could leverage large language model (LLM) capabilities. However, the scarcity of challenging financial datasets, particularly for Japanese financial data, impedes academic innovation in financial analytics. As LLMs advance, this lack of accessible research resources increasingly hinders their development and evaluation in this specialized domain. To address this gap, we introduce EDINET-Bench, an open-source Japanese financial benchmark designed to evaluate the performance of LLMs on challenging financial tasks including accounting fraud detection, earnings forecasting, and industry prediction. EDINET-Bench is constructed by downloading annual reports from the past 10 years from Japan's Electronic Disclosure for Investors' NETwork (EDINET) and automatically assigning labels corresponding to each evaluation task. Our experiments reveal that even state-of-the-art LLMs struggle, performing only slightly better than logistic regression in binary classification for fraud detection and earnings forecasting. These results highlight significant challenges in applying LLMs to real-world financial applications and underscore the need for domain-specific adaptation. Our dataset, benchmark construction code, and evaluation code is publicly available to facilitate future research in finance with LLMs.