Follow the Flow: Fine-grained Flowchart Attribution with Neurosymbolic Agents
This addresses the problem of unreliable automated flowchart processing in critical domains like logistics and health, though it is incremental as it builds on existing neurosymbolic and graph-based methods.
The paper tackles the challenge of interpreting flowcharts with LLMs, which often hallucinate connections, by introducing a neurosymbolic agent that performs fine-grained attribution to improve verifiability and explainability, achieving a 10-14% performance gain over baselines on a new benchmark.
Flowcharts are a critical tool for visualizing decision-making processes. However, their non-linear structure and complex visual-textual relationships make it challenging to interpret them using LLMs, as vision-language models frequently hallucinate nonexistent connections and decision paths when analyzing these diagrams. This leads to compromised reliability for automated flowchart processing in critical domains such as logistics, health, and engineering. We introduce the task of Fine-grained Flowchart Attribution, which traces specific components grounding a flowchart referring LLM response. Flowchart Attribution ensures the verifiability of LLM predictions and improves explainability by linking generated responses to the flowchart's structure. We propose FlowPathAgent, a neurosymbolic agent that performs fine-grained post hoc attribution through graph-based reasoning. It first segments the flowchart, then converts it into a structured symbolic graph, and then employs an agentic approach to dynamically interact with the graph, to generate attribution paths. Additionally, we present FlowExplainBench, a novel benchmark for evaluating flowchart attributions across diverse styles, domains, and question types. Experimental results show that FlowPathAgent mitigates visual hallucinations in LLM answers over flowchart QA, outperforming strong baselines by 10-14% on our proposed FlowExplainBench dataset.