Responses Fall Short of Understanding: Revealing the Gap between Internal Representations and Responses in Visual Document Understanding
This addresses a critical issue for researchers and practitioners in AI by highlighting a potential misalignment in model evaluation and offering a method to enhance performance, though it is incremental as it builds on existing fine-tuning techniques.
The paper tackles the problem that large vision-language models (LVLMs) may not accurately reflect their internal understanding in generated responses for visual document understanding (VDU) tasks, revealing a gap between internal representations and responses and showing that fine-tuning intermediate layers improves both linear probing accuracy and response accuracy.
Visual document understanding (VDU) is a challenging task for large vision language models (LVLMs), requiring the integration of visual perception, text recognition, and reasoning over structured layouts. Although recent LVLMs have shown progress on VDU benchmarks, their performance is typically evaluated based on generated responses, which may not necessarily reflect whether the model has actually captured the required information internally. In this paper, we investigate how information required to solve VDU tasks is represented across different layers of LLMs within LVLMs using linear probing. Our study reveals that (1) there is a clear gap between internal representations and generated responses, and (2) information required to solve the task is often encoded more linearly from intermediate layers than from the final layer. Motivated by these findings, we explore fine-tuning strategies that target intermediate layers. Experiments show that fine-tuning intermediate layers improves both linear probing accuracy and response accuracy while narrowing the gap.