HiRes-LLaVA: Restoring Fragmentation Input in High-Resolution Large Vision-Language Models
This work addresses a specific bottleneck in high-resolution vision-language models for tasks requiring cross-patch context and position awareness, offering an incremental improvement over existing slicing methods.
The paper tackles the problem of fragmentation in high-resolution vision-language models caused by sliding window slicing, which loses contextual and spatial information, by introducing HiRes-LLaVA, a framework that restores original input structure and reduces training costs, achieving superior performance on benchmarks including a new EntityGrid-QA benchmark for document-oriented tasks.
High-resolution inputs enable Large Vision-Language Models (LVLMs) to discern finer visual details, enhancing their comprehension capabilities. To reduce the training and computation costs caused by high-resolution input, one promising direction is to use sliding windows to slice the input into uniform patches, each matching the input size of the well-trained vision encoder. Although efficient, this slicing strategy leads to the fragmentation of original input, i.e., the continuity of contextual information and spatial geometry is lost across patches, adversely affecting performance in cross-patch context perception and position-specific tasks. To overcome these shortcomings, we introduce HiRes-LLaVA, a novel framework designed to efficiently process any size of high-resolution input without altering the original contextual and geometric information. HiRes-LLaVA comprises two innovative components: (i) a SliceRestore adapter that reconstructs sliced patches into their original form, efficiently extracting both global and local features via down-up-sampling and convolution layers, and (ii) a Self-Mining Sampler to compresses the vision tokens based on themselves, preserving the original context and positional information while reducing training overhead. To assess the ability of handling context fragmentation, we construct a new benchmark, EntityGrid-QA, consisting of edge-related and position-related tasks. Our comprehensive experiments demonstrate the superiority of HiRes-LLaVA on both existing public benchmarks and on EntityGrid-QA, particularly on document-oriented tasks, establishing new standards for handling high-resolution inputs.