Finding NeMo: Negative-mined Mosaic Augmentation for Referring Image Segmentation
This addresses the bottleneck in data for Referring Image Segmentation, making it more robust in complex scenarios, though it is an incremental improvement over existing methods.
The paper tackles the performance gap in Referring Image Segmentation between easy and hard scenarios by proposing Negative-mined Mosaic Augmentation (NeMo), a data augmentation method that creates challenging training samples using negative images curated by a pretrained model like CLIP, resulting in consistent improvements across various datasets and models.
Referring Image Segmentation is a comprehensive task to segment an object referred by a textual query from an image. In nature, the level of difficulty in this task is affected by the existence of similar objects and the complexity of the referring expression. Recent RIS models still show a significant performance gap between easy and hard scenarios. We pose that the bottleneck exists in the data, and propose a simple but powerful data augmentation method, Negative-mined Mosaic Augmentation (NeMo). This method augments a training image into a mosaic with three other negative images carefully curated by a pretrained multimodal alignment model, e.g., CLIP, to make the sample more challenging. We discover that it is critical to properly adjust the difficulty level, neither too ambiguous nor too trivial. The augmented training data encourages the RIS model to recognize subtle differences and relationships between similar visual entities and to concretely understand the whole expression to locate the right target better. Our approach shows consistent improvements on various datasets and models, verified by extensive experiments.