Spatially Grounded Explanations in Vision Language Models for Document Visual Question Answering
This work addresses the need for interpretability in vision language models for document analysis, offering a model-agnostic solution that enhances transparency without fine-tuning, though it is incremental as it builds on existing methods for grounding explanations.
The paper tackles the problem of generating transparent and reproducible explanations in document visual question answering by introducing EaGERS, a training-free pipeline that grounds natural language rationales to spatial regions and restricts responses to relevant areas, resulting in improved exact match accuracy and Average Normalized Levenshtein Similarity metrics on the DocVQA dataset.
We introduce EaGERS, a fully training-free and model-agnostic pipeline that (1) generates natural language rationales via a vision language model, (2) grounds these rationales to spatial sub-regions by computing multimodal embedding similarities over a configurable grid with majority voting, and (3) restricts the generation of responses only from the relevant regions selected in the masked image. Experiments on the DocVQA dataset demonstrate that our best configuration not only outperforms the base model on exact match accuracy and Average Normalized Levenshtein Similarity metrics but also enhances transparency and reproducibility in DocVQA without additional model fine-tuning.