Structured Attention Matters to Multimodal LLMs in Document Understanding
This work addresses document understanding challenges for MLLMs, offering a novel method to improve performance, though it is incremental as it builds on existing attention mechanisms without architectural innovation.
The paper tackled the problem of how input format affects multimodal large language models (MLLMs) in document understanding, finding that raw OCR text impairs performance due to attention dispersion, and proposed a structure-preserving LaTex-based approach that significantly enhanced question answering performance across diverse documents without model changes.
Document understanding remains a significant challenge for multimodal large language models (MLLMs). While previous research has primarily focused on locating evidence pages through precise multimodal queries, our work investigates a fundamental yet overlooked aspect: how input format influences document comprehension performance. Through systematic analysis, we discover that raw OCR text often impairs rather than improves MLLMs' performance, which is a counterintuitive finding we attribute to attention dispersion and structure loss. To further substantiate our hypothesis, we propose a novel structure-preserving approach that encodes document elements using the LaTex paradigm, maintaining the hierarchical organization and spatial relationships critical for comprehension. Our attention analysis reveals that structured text induces structured attention patterns on both textual and visual content, directing models to focus on semantically meaningful regions while reducing attention waste. This approach significantly enhances MLLMs' document question answering performance across diverse document types without requiring architectural modifications or additional training.