FinQAPT: Empowering Financial Decisions with End-to-End LLM-driven Question Answering Pipeline
This work addresses financial decision-making for analysts by providing a pipeline to process documents, though it is incremental with performance drops at the pipeline level.
The authors tackled the problem of financial decision-making by developing FinQAPT, an end-to-end LLM-driven pipeline for analyzing financial documents, achieving state-of-the-art accuracy of 80.6% on the FinQA dataset at the module level.
Financial decision-making hinges on the analysis of relevant information embedded in the enormous volume of documents in the financial domain. To address this challenge, we developed FinQAPT, an end-to-end pipeline that streamlines the identification of relevant financial reports based on a query, extracts pertinent context, and leverages Large Language Models (LLMs) to perform downstream tasks. To evaluate the pipeline, we experimented with various techniques to optimize the performance of each module using the FinQA dataset. We introduced a novel clustering-based negative sampling technique to enhance context extraction and a novel prompting method called Dynamic N-shot Prompting to boost the numerical question-answering capabilities of LLMs. At the module level, we achieved state-of-the-art accuracy on FinQA, attaining an accuracy of 80.6%. However, at the pipeline level, we observed decreased performance due to challenges in extracting relevant context from financial reports. We conducted a detailed error analysis of each module and the end-to-end pipeline, pinpointing specific challenges that must be addressed to develop a robust solution for handling complex financial tasks.