Cooperative Semantic Segmentation and Image Restoration in Adverse Environmental Conditions
This work addresses the performance drop of semantic segmentation in adverse conditions, which is a practical issue for applications like autonomous driving, but it is incremental as it builds on existing methods by combining them in a holistic manner.
The paper tackles the problem of semantic segmentation and image restoration in adverse environmental conditions by proposing a cooperative deep-learning framework that integrates both tasks, resulting in improved segmentation accuracy and enhanced perceptual quality and structural similarity of restored images.
Most state-of-the-art semantic segmentation approaches only achieve high accuracy in good conditions. In practically-common but less-discussed adverse environmental conditions, their performance can decrease enormously. Existing studies usually cast the handling of segmentation in adverse conditions as a separate post-processing step after signal restoration, making the segmentation performance largely depend on the quality of restoration. In this paper, we propose a novel deep-learning framework to tackle semantic segmentation and image restoration in adverse environmental conditions in a holistic manner. The proposed approach contains two components: Semantically-Guided Adaptation, which exploits semantic information from degraded images to refine the segmentation; and Exemplar-Guided Synthesis, which restores images from semantic label maps given degraded exemplars as the guidance. Our method cooperatively leverages the complementarity and interdependence of low-level restoration and high-level segmentation in adverse environmental conditions. Extensive experiments on various datasets demonstrate that our approach can not only improve the accuracy of semantic segmentation with degradation cues, but also boost the perceptual quality and structural similarity of image restoration with semantic guidance.