MMNet: Multi-Mask Network for Referring Image Segmentation
This addresses the problem of accurately segmenting objects based on natural language descriptions for computer vision applications, representing an incremental improvement over prior methods.
The paper tackles the challenge of referring image segmentation, where diverse objects and unrestricted language expressions introduce uncertainty, by proposing MMNet, which generates multiple segmentation masks and combines them via weighted sum, achieving superior performance on RefCOCO, RefCOCO+, and G-Ref datasets without post-processing.
Referring image segmentation aims to segment an object referred to by natural language expression from an image. However, this task is challenging due to the distinct data properties between text and image, and the randomness introduced by diverse objects and unrestricted language expression. Most of previous work focus on improving cross-modal feature fusion while not fully addressing the inherent uncertainty caused by diverse objects and unrestricted language. To tackle these problems, we propose an end-to-end Multi-Mask Network for referring image segmentation(MMNet). we first combine picture and language and then employ an attention mechanism to generate multiple queries that represent different aspects of the language expression. We then utilize these queries to produce a series of corresponding segmentation masks, assigning a score to each mask that reflects its importance. The final result is obtained through the weighted sum of all masks, which greatly reduces the randomness of the language expression. Our proposed framework demonstrates superior performance compared to state-of-the-art approaches on the two most commonly used datasets, RefCOCO, RefCOCO+ and G-Ref, without the need for any post-processing. This further validates the efficacy of our proposed framework.