CalibCLIP: Contextual Calibration of Dominant Semantics for Text-Driven Image Retrieval
This addresses a structural limitation in VLMs for image retrieval, offering a practical solution to enhance retrieval accuracy, though it is incremental as it builds on existing CLIP-based models.
The paper tackles the problem of dominant tokens in Visual Language Models suppressing discriminative features in text-driven image retrieval by introducing CalibCLIP, a training-free method that calibrates these tokens, resulting in consistent improvements across seven benchmarks in three retrieval tasks.
Existing Visual Language Models (VLMs) suffer structural limitations where a few low contribution tokens may excessively capture global semantics, dominating the information aggregation process and suppressing the discriminative features in text-driven image retrieval tasks. To address this, we introduce \textbf{CalibCLIP}, a training-free method designed to calibrate the suppressive effect of dominant tokens. Specifically, in the visual space, we propose the Contrastive Visual Enhancer (CVE), which decouples visual features into target and low information regions. Subsequently, it identifies dominant tokens and dynamically suppresses their representations.In the textual space, we introduce the Discriminative Concept Calibrator (DCC), which aims to differentiate between general and discriminative concepts within the text query. By mitigating the challenges posed by generic concepts and improving the representations of discriminative concepts, DCC strengthens the differentiation among similar samples. Finally, extensive experiments demonstrate consistent improvements across seven benchmarks spanning three image retrieval tasks, underscoring the effectiveness of CalibCLIP. Code is available at: https://github.com/kangbin98/CalibCLIP