REPAIR: Rank Correlation and Noisy Pair Half-replacing with Memory for Noisy Correspondence
This work addresses performance degradation in cross-modal matching due to noisy data, which is a common issue in multimodal applications where precise annotations are costly, though it appears incremental as it builds on existing noisy correspondence methods.
The paper tackles the problem of noisy correspondence in cross-modal matching by proposing REPAIR, a framework that uses a memory bank to estimate soft labels and replace mismatched features, achieving improved performance on datasets like Flickr30K, MSCOCO, and CC152K with synthetic and real-world noise.
The presence of noise in acquired data invariably leads to performance degradation in cross-modal matching. Unfortunately, obtaining precise annotations in the multimodal field is expensive, which has prompted some methods to tackle the mismatched data pair issue in cross-modal matching contexts, termed as noisy correspondence. However, most of these existing noisy correspondence methods exhibit the following limitations: a) the problem of self-reinforcing error accumulation, and b) improper handling of noisy data pair. To tackle the two problems, we propose a generalized framework termed as Rank corrElation and noisy Pair hAlf-replacing wIth memoRy (REPAIR), which benefits from maintaining a memory bank for features of matched pairs. Specifically, we calculate the distances between the features in the memory bank and those of the target pair for each respective modality, and use the rank correlation of these two sets of distances to estimate the soft correspondence label of the target pair. Estimating soft correspondence based on memory bank features rather than using a similarity network can avoid the accumulation of errors due to incorrect network identifications. For pairs that are completely mismatched, REPAIR searches the memory bank for the most matching feature to replace one feature of one modality, instead of using the original pair directly or merely discarding the mismatched pair. We conduct experiments on three cross-modal datasets, i.e., Flickr30K, MSCOCO, and CC152K, proving the effectiveness and robustness of our REPAIR on synthetic and real-world noise.