PRSeg: A Lightweight Patch Rotate MLP Decoder for Semantic Segmentation
This work addresses a bottleneck in lightweight semantic segmentation decoders, which is an incremental improvement for computer vision applications.
The authors tackled the problem of limited receptive fields in lightweight MLP-based decoders for semantic segmentation by proposing a parametric-free patch rotate operation to reorganize pixels spatially, resulting in a novel network called PRSeg that achieves effectiveness as proven by extensive experiments on ADE20K, Cityscapes, and COCO-Stuff 10K datasets.
The lightweight MLP-based decoder has become increasingly promising for semantic segmentation. However, the channel-wise MLP cannot expand the receptive fields, lacking the context modeling capacity, which is critical to semantic segmentation. In this paper, we propose a parametric-free patch rotate operation to reorganize the pixels spatially. It first divides the feature map into multiple groups and then rotates the patches within each group. Based on the proposed patch rotate operation, we design a novel segmentation network, named PRSeg, which includes an off-the-shelf backbone and a lightweight Patch Rotate MLP decoder containing multiple Dynamic Patch Rotate Blocks (DPR-Blocks). In each DPR-Block, the fully connected layer is performed following a Patch Rotate Module (PRM) to exchange spatial information between pixels. Specifically, in PRM, the feature map is first split into the reserved part and rotated part along the channel dimension according to the predicted probability of the Dynamic Channel Selection Module (DCSM), and our proposed patch rotate operation is only performed on the rotated part. Extensive experiments on ADE20K, Cityscapes and COCO-Stuff 10K datasets prove the effectiveness of our approach. We expect that our PRSeg can promote the development of MLP-based decoder in semantic segmentation.