Improved LLM Agents for Financial Document Question Answering
This work addresses a domain-specific challenge in financial document analysis, offering incremental improvements over existing methods.
The paper tackled the problem of LLMs struggling with numerical question answering on financial documents by proposing an improved critic agent and a calculator agent, which together outperformed the previous state-of-the-art program-of-thought approach and enhanced safety.
Large language models (LLMs) have shown impressive capabilities on numerous natural language processing tasks. However, LLMs still struggle with numerical question answering for financial documents that include tabular and textual data. Recent works have showed the effectiveness of critic agents (i.e., self-correction) for this task given oracle labels. Building upon this framework, this paper examines the effectiveness of the traditional critic agent when oracle labels are not available, and show, through experiments, that this critic agent's performance deteriorates in this scenario. With this in mind, we present an improved critic agent, along with the calculator agent which outperforms the previous state-of-the-art approach (program-of-thought) and is safer. Furthermore, we investigate how our agents interact with each other, and how this interaction affects their performance.