OmniSegmentor: A Flexible Multi-Modal Learning Framework for Semantic Segmentation
This addresses the problem of robust semantic segmentation for computer vision applications by providing a universal multi-modal pretraining framework, though it is incremental as it builds on existing multi-modal representation learning.
The paper tackles the lack of a flexible pretrain-and-finetune pipeline for multi-modal semantic segmentation by proposing OmniSegmentor, which achieves new state-of-the-art records on multiple datasets including NYU Depthv2 and KITTI-360.
Recent research on representation learning has proved the merits of multi-modal clues for robust semantic segmentation. Nevertheless, a flexible pretrain-and-finetune pipeline for multiple visual modalities remains unexplored. In this paper, we propose a novel multi-modal learning framework, termed OmniSegmentor. It has two key innovations: 1) Based on ImageNet, we assemble a large-scale dataset for multi-modal pretraining, called ImageNeXt, which contains five popular visual modalities. 2) We provide an efficient pretraining manner to endow the model with the capacity to encode different modality information in the ImageNeXt. For the first time, we introduce a universal multi-modal pretraining framework that consistently amplifies the model's perceptual capabilities across various scenarios, regardless of the arbitrary combination of the involved modalities. Remarkably, our OmniSegmentor achieves new state-of-the-art records on a wide range of multi-modal semantic segmentation datasets, including NYU Depthv2, EventScape, MFNet, DeLiVER, SUNRGBD, and KITTI-360.