TALENT: Table VQA via Augmented Language-Enhanced Natural-text Transcription
This addresses the need for efficient Table VQA systems, especially for mobile deployment, by reducing computational requirements while maintaining accuracy.
The paper tackles the problem of Table Visual Question Answering (Table VQA) by proposing TALENT, a lightweight framework that uses a small vision-language model for OCR and natural language narration, combined with a large language model for reasoning, enabling performance matching or surpassing a single large VLM at lower computational cost on public datasets and a new challenging dataset, ReTabVQA.
Table Visual Question Answering (Table VQA) is typically addressed by large vision-language models (VLMs). While such models can answer directly from images, they often miss fine-grained details unless scaled to very large sizes, which are computationally prohibitive, especially for mobile deployment. A lighter alternative is to have a small VLM perform OCR and then use a large language model (LLM) to reason over structured outputs such as Markdown tables. However, these representations are not naturally optimized for LLMs and still introduce substantial errors. We propose TALENT (Table VQA via Augmented Language-Enhanced Natural-text Transcription), a lightweight framework that leverages dual representations of tables. TALENT prompts a small VLM to produce both OCR text and natural language narration, then combines them with the question for reasoning by an LLM. This reframes Table VQA as an LLM-centric multimodal reasoning task, where the VLM serves as a perception-narration module rather than a monolithic solver. Additionally, we construct ReTabVQA, a more challenging Table VQA dataset requiring multi-step quantitative reasoning over table images. Experiments show that TALENT enables a small VLM-LLM combination to match or surpass a single large VLM at significantly lower computational cost on both public datasets and ReTabVQA.