U-shaped Multi-granularity Learning for Vision-Language Models
For researchers working on vision-language models for dense prediction tasks, this provides a new framework that balances fine-grained and global semantics with minimal overhead.
UPrompt introduces a U-shaped multi-granularity prompt learning framework for vision-language models that addresses the granularity dilemma between global and local prompts, achieving state-of-the-art results on dense prediction tasks with up to 7.3 rSum improvement on MSCOCO and 5.09% gain in base-to-novel generalization.
The prompt learning paradigm for vision-language models is effective yet faces a granularity dilemma: global prompts lack fine-grained semantic awareness, while local prompts ignore contextual associations, limiting cross-task generalization. This dilemma exists in dense prediction tasks. Inspired by U-Net, which unifies multi-level representations across granularities, we propose UPrompt, a U-shaped multi-granularity prompt learning framework for vision-language models. Similar to how U-Net integrates fine and coarse features through symmetric encoder-decoder pathways with cross-level connections, UPrompt constructs parallel multi-granularity representations in both visual and textual modalities, where coarse-to-fine cascaded enhancement propagates global context to refine local details, while fine-to-coarse hierarchical supervision ensures semantic consistency across scales. Extensive experiments on 17 benchmarks validate our effectiveness. UPrompt outperforms MAMET and VPKE by 4.1 and 7.3 rSum on MSCOCO, surpasses CoCoA-Mix by 5.09% in base-to-novel generalization, while maintaining competitive performance with minimal overhead (coarse-grained) and matching PSRC with 1/3 cost (medium-grained).