MulCLIP: A Multi-level Alignment Framework for Enhancing Fine-grained Long-context CLIP
This addresses the problem of fine-grained vision-language alignment for lengthy text descriptions, which is important for real-world applications, though it appears incremental relative to existing region-proposal methods.
The paper tackles CLIP's difficulty with lengthy, detailed descriptions by introducing MulCLIP, a multi-level alignment framework that improves fine-grained understanding through token reconstruction and subcaption-aggregated patch alignment, achieving consistent performance gains across diverse benchmarks.
Vision-language models like CLIP show impressive ability to align images and text, but their training on short, concise captions makes them struggle with lengthy, detailed descriptions. Recent advances mitigate this challenge by leveraging region-proposal information to map visual regions with corresponding sentences from lengthy captions, yet incurring notable deployment costs. We introduce MulCLIP, a novel end-to-end multi-level alignment framework that bridges natural long-text structures with image components. MulCLIP first preserves global contrastive alignment between images and both summary and long captions, while extending positional embeddings for longer text sequences. To further enhance fine-grained understanding, we propose two novel strategies: (1) a token reconstruction alignment over locally calibrated features to strengthen semantic connections between words and image patches, and (2) a subcaption-aggregated patch alignment that automatically extracts and aggregates context-rich patches for each subcaption. Experimental results across diverse benchmarks demonstrate our method consistently improves downstream performance, while ablation studies confirm its multi-scale alignment is the key factor driving better fine-grained capability than region-proposal-assisted approaches, making it particularly suitable for diverse real-world applications.