Dual Convolutional LSTM Network for Referring Image Segmentation
This work addresses the problem of segmenting objects in images based on natural language queries, which is important for applications in computer vision and human-computer interaction, but it appears incremental as it builds on existing ConvLSTM and attention mechanisms.
The authors tackled referring image segmentation by proposing a dual convolutional LSTM network that integrates visual and linguistic features with attention, achieving superior performance on four challenging datasets compared to state-of-the-art methods.
We consider referring image segmentation. It is a problem at the intersection of computer vision and natural language understanding. Given an input image and a referring expression in the form of a natural language sentence, the goal is to segment the object of interest in the image referred by the linguistic query. To this end, we propose a dual convolutional LSTM (ConvLSTM) network to tackle this problem. Our model consists of an encoder network and a decoder network, where ConvLSTM is used in both encoder and decoder networks to capture spatial and sequential information. The encoder network extracts visual and linguistic features for each word in the expression sentence, and adopts an attention mechanism to focus on words that are more informative in the multimodal interaction. The decoder network integrates the features generated by the encoder network at multiple levels as its input and produces the final precise segmentation mask. Experimental results on four challenging datasets demonstrate that the proposed network achieves superior segmentation performance compared with other state-of-the-art methods.