Recalling Holistic Information for Semantic Segmentation
This work addresses semantic segmentation for computer vision applications, offering an incremental improvement by combining holistic and local features.
The paper tackles the problem of noisy pixel labels in semantic segmentation by proposing a two-stream neural network that integrates high-recall holistic image information with local details, achieving state-of-the-art performance on PASCAL-Context and NYUDv2 datasets.
Semantic segmentation requires a detailed labeling of image pixels by object category. Information derived from local image patches is necessary to describe the detailed shape of individual objects. However, this information is ambiguous and can result in noisy labels. Global inference of image content can instead capture the general semantic concepts present. We advocate that high-recall holistic inference of image concepts provides valuable information for detailed pixel labeling. We build a two-stream neural network architecture that facilitates information flow from holistic information to local pixels, while keeping common image features shared among the low-level layers of both the holistic analysis and segmentation branches. We empirically evaluate our network on four standard semantic segmentation datasets. Our network obtains state-of-the-art performance on PASCAL-Context and NYUDv2, and ablation studies verify its effectiveness on ADE20K and SIFT-Flow.