FinS-Pilot: A Benchmark for Online Financial RAG System
This addresses the problem of evaluating financial RAG systems for researchers and practitioners, but it is incremental as it builds on existing RAG and benchmark methodologies by focusing on the financial domain.
The authors tackled the lack of specialized benchmarks for evaluating retrieval-augmented generation (RAG) systems in online financial applications by introducing FinS-Pilot, a benchmark constructed from real-world financial assistant interactions that incorporates real-time API and text data, demonstrating its effectiveness in identifying suitable models for financial use through systematic experiments with multiple leading Chinese LLMs.
Large language models (LLMs) have demonstrated remarkable capabilities across various professional domains, with their performance typically evaluated through standardized benchmarks. In the financial field, the stringent demands for professional accuracy and real-time data processing often necessitate the use of retrieval-augmented generation (RAG) techniques. However, the development of financial RAG benchmarks has been constrained by data confidentiality issues and the lack of dynamic data integration. To address this issue, we introduce FinS-Pilot, a novel benchmark for evaluating RAG systems in online financial applications. Constructed from real-world financial assistant interactions, our benchmark incorporates both real-time API data and text data, organized through an intent classification framework covering critical financial domains. The benchmark enables comprehensive evaluation of financial assistants' capabilities in handling both static knowledge and time-sensitive market information.Through systematic experiments with multiple Chinese leading LLMs, we demonstrate FinS-Pilot's effectiveness in identifying models suitable for financial applications while addressing the current gap in specialized evaluation tools for the financial domain. Our work contributes both a practical evaluation framework and a curated dataset to advance research in financial NLP systems. The code and dataset are accessible on GitHub.