Benchmarking Robustness of Text-Image Composed Retrieval
This work addresses the robustness problem for researchers and practitioners in multimodal AI by providing systematic evaluation tools, though it is incremental as it focuses on benchmarking rather than developing new methods.
The paper tackles the lack of robustness studies in text-image composed retrieval by establishing three new benchmarks to analyze performance against natural corruptions in vision and text, and to probe textual understanding, introducing datasets like CIRR-C and FashionIQ-C with up to 15 visual and 7 textual corruptions.
Text-image composed retrieval aims to retrieve the target image through the composed query, which is specified in the form of an image plus some text that describes desired modifications to the input image. It has recently attracted attention due to its ability to leverage both information-rich images and concise language to precisely express the requirements for target images. However, the robustness of these approaches against real-world corruptions or further text understanding has never been studied. In this paper, we perform the first robustness study and establish three new diversified benchmarks for systematic analysis of text-image composed retrieval against natural corruptions in both vision and text and further probe textural understanding. For natural corruption analysis, we introduce two new large-scale benchmark datasets, CIRR-C and FashionIQ-C for testing in open domain and fashion domain respectively, both of which apply 15 visual corruptions and 7 textural corruptions. For textural understanding analysis, we introduce a new diagnostic dataset CIRR-D by expanding the original raw data with synthetic data, which contains modified text to better probe textual understanding ability including numerical variation, attribute variation, object removal, background variation, and fine-grained evaluation. The code and benchmark datasets are available at https://github.com/SunTongtongtong/Benchmark-Robustness-Text-Image-Compose-Retrieval.