Encoder Fusion Network with Co-Attention Embedding for Referring Image Segmentation
This work addresses the challenge of accurately segmenting objects based on natural language descriptions, which is important for applications like human-computer interaction, but it appears incremental as it builds on existing multi-modal fusion methods.
The paper tackles the problem of referring image segmentation by proposing an encoder fusion network with co-attention embedding to integrate language and vision features more effectively, achieving state-of-the-art performance on four benchmark datasets without post-processing.
Recently, referring image segmentation has aroused widespread interest. Previous methods perform the multi-modal fusion between language and vision at the decoding side of the network. And, linguistic feature interacts with visual feature of each scale separately, which ignores the continuous guidance of language to multi-scale visual features. In this work, we propose an encoder fusion network (EFN), which transforms the visual encoder into a multi-modal feature learning network, and uses language to refine the multi-modal features progressively. Moreover, a co-attention mechanism is embedded in the EFN to realize the parallel update of multi-modal features, which can promote the consistent of the cross-modal information representation in the semantic space. Finally, we propose a boundary enhancement module (BEM) to make the network pay more attention to the fine structure. The experiment results on four benchmark datasets demonstrate that the proposed approach achieves the state-of-the-art performance under different evaluation metrics without any post-processing.