Case-Based Reasoning Approach for Solving Financial Question Answering
This work addresses the challenge of improving reasoning accuracy in financial AI systems, representing an incremental advancement by applying an existing AI paradigm to a specific bottleneck in a domain-specific dataset.
The paper tackled the problem of numerical reasoning in financial question answering by addressing errors in program generation, proposing a case-based reasoning approach that retrieves similar cases to guide answer generation, achieving competitive performance on the FinQA dataset and improving handling of complex multi-step programs.
Measuring a machine's understanding of human language often involves assessing its reasoning skills, i.e. logical process of deriving answers to questions. While recent language models have shown remarkable proficiency in text based tasks, their efficacy in complex reasoning problems involving heterogeneous information such as text, tables, and numbers remain uncertain. Addressing this gap, FinQA introduced a numerical reasoning dataset for financial documents and simultaneously proposed a program generation approach . Our investigation reveals that half of the errors (48%) stem from incorrect operations being generated. To address this issue, we propose a novel approach to tackle numerical reasoning problems using case based reasoning (CBR), an artificial intelligence paradigm that provides problem solving guidance by offering similar cases (i.e. similar questions and corresponding logical programs). Our model retrieves relevant cases to address a given question, and then generates an answer based on the retrieved cases and contextual information. Through experiments on the FinQA dataset, we demonstrate competitive performance of our approach and additionally show that by expanding case repository, we can help solving complex multi step programs which FinQA showed weakness of.