Modulating Bottom-Up and Top-Down Visual Processing via Language-Conditional Filters
This work addresses the challenge of effectively combining language and vision for dense-prediction tasks, offering an incremental improvement over existing methods.
The authors tackled the problem of integrating linguistic and perceptual processing in multimodal tasks by proposing that language should condition both bottom-up and top-down visual processing, not just top-down attention. Their experiments on referring expression segmentation and language-guided image colorization showed that this approach leads to better results and achieves competitive performance.
How to best integrate linguistic and perceptual processing in multi-modal tasks that involve language and vision is an important open problem. In this work, we argue that the common practice of using language in a top-down manner, to direct visual attention over high-level visual features, may not be optimal. We hypothesize that the use of language to also condition the bottom-up processing from pixels to high-level features can provide benefits to the overall performance. To support our claim, we propose a U-Net-based model and perform experiments on two language-vision dense-prediction tasks: referring expression segmentation and language-guided image colorization. We compare results where either one or both of the top-down and bottom-up visual branches are conditioned on language. Our experiments reveal that using language to control the filters for bottom-up visual processing in addition to top-down attention leads to better results on both tasks and achieves competitive performance. Our linguistic analysis suggests that bottom-up conditioning improves segmentation of objects especially when input text refers to low-level visual concepts. Code is available at https://github.com/ilkerkesen/bvpr.