FBCIR: Balancing Cross-Modal Focuses in Composed Image Retrieval
This work addresses robustness issues in multi-modal retrieval for AI systems, though it is incremental as it builds on existing models with data augmentation.
The paper tackled the problem of focus imbalances in composed image retrieval models, where models disproportionately attend to one modality, leading to degraded accuracy in challenging scenarios; they proposed a data augmentation workflow that improved performance in hard negative cases while maintaining standard benchmark capabilities.
Composed image retrieval (CIR) requires multi-modal models to jointly reason over visual content and semantic modifications presented in text-image input pairs. While current CIR models achieve strong performance on common benchmark cases, their accuracies often degrades in more challenging scenarios where negative candidates are semantically aligned with the query image or text. In this paper, we attribute this degradation to focus imbalances, where models disproportionately attend to one modality while neglecting the other. To validate this claim, we propose FBCIR, a multi-modal focus interpretation method that identifies the most crucial visual and textual input components to a model's retrieval decisions. Using FBCIR, we report that focus imbalances are prevalent in existing CIR models, especially under hard negative settings. Building on the analyses, we further propose a CIR data augmentation workflow that facilitates existing CIR datasets with curated hard negatives designed to encourage balanced cross-modal reasoning. Extensive experiments across multiple CIR models demonstrate that the proposed augmentation consistently improves performance in challenging cases, while maintaining their capabilities on standard benchmarks. Together, our interpretation method and data augmentation workflow provide a new perspective on CIR model diagnosis and robustness improvements.