PowerCLIP: Powerset Alignment for Contrastive Pre-Training
This addresses the problem of fine-grained compositional understanding in vision-language models for researchers and practitioners, representing an incremental improvement over existing methods.
The paper tackles the challenge of capturing compositional semantics spanning multiple image regions in vision-language pre-training by proposing PowerCLIP, a framework that uses powerset alignment to optimize region-to-phrase alignments, resulting in outperforming state-of-the-art methods in zero-shot classification and retrieval tasks.
Contrastive vision-language pre-training frameworks such as CLIP have demonstrated impressive zero-shot performance across a range of vision-language tasks. Recent studies have shown that aligning individual text tokens with specific image patches or regions enhances fine-grained compositional understanding. However, it remains challenging to capture compositional semantics that span multiple image regions. To address this limitation, we propose PowerCLIP, a novel contrastive pre-training framework enhanced by powerset alignment, which exhaustively optimizes region-to-phrase alignments by minimizing the loss defined between powersets of image regions and textual parse trees. Since the naive powerset construction incurs exponential computational cost due to the combinatorial explosion in the number of region subsets, we introduce efficient non-linear aggregators (NLAs) that reduce complexity from O(2^M) to O(M) with respect to the number of regions M, while approximating the exact loss value with arbitrary precision. Our extensive experiments demonstrate that PowerCLIP outperforms state-of-the-art methods in zero-shot classification and retrieval tasks, underscoring the compositionality and robustness of our approach. Our code will be made publicly available.