LLM Augmentations to support Analytical Reasoning over Multiple Documents
This addresses the challenge for intelligence analysts who need to process large dossiers to uncover connections and plans, but it is incremental as it builds on existing LLM capabilities with a new augmentation.
The paper tackled the problem of using large language models (LLMs) to enhance analytical reasoning over multiple documents for intelligence analysis, finding that current LLMs are inadequate and proposing an architecture with dynamic evidence trees to improve performance.
Building on their demonstrated ability to perform a variety of tasks, we investigate the application of large language models (LLMs) to enhance in-depth analytical reasoning within the context of intelligence analysis. Intelligence analysts typically work with massive dossiers to draw connections between seemingly unrelated entities, and uncover adversaries' plans and motives. We explore if and how LLMs can be helpful to analysts for this task and develop an architecture to augment the capabilities of an LLM with a memory module called dynamic evidence trees (DETs) to develop and track multiple investigation threads. Through extensive experiments on multiple datasets, we highlight how LLMs, as-is, are still inadequate to support intelligence analysts and offer recommendations to improve LLMs for such intricate reasoning applications.