ChartCitor: Multi-Agent Framework for Fine-Grained Chart Visual Attribution
This addresses the issue of limited visual-semantic context and complex alignment in chart QA for professionals, though it appears incremental as it builds on existing multi-agent and retrieval methods.
The paper tackles the problem of LLMs generating unverified hallucinated responses in chart question-answering by introducing ChartCitor, a multi-agent framework that provides fine-grained bounding box citations for supporting evidence, outperforming existing baselines across different chart types.
Large Language Models (LLMs) can perform chart question-answering tasks but often generate unverified hallucinated responses. Existing answer attribution methods struggle to ground responses in source charts due to limited visual-semantic context, complex visual-text alignment requirements, and difficulties in bounding box prediction across complex layouts. We present ChartCitor, a multi-agent framework that provides fine-grained bounding box citations by identifying supporting evidence within chart images. The system orchestrates LLM agents to perform chart-to-table extraction, answer reformulation, table augmentation, evidence retrieval through pre-filtering and re-ranking, and table-to-chart mapping. ChartCitor outperforms existing baselines across different chart types. Qualitative user studies show that ChartCitor helps increase user trust in Generative AI by providing enhanced explainability for LLM-assisted chart QA and enables professionals to be more productive.