Robust Video Content Alignment and Compensation for Rain Removal in a CNN Framework
This addresses the problem of improving outdoor vision system robustness in challenging conditions like fast camera motion and torrential rain, representing a strong specific gain rather than a broad paradigm shift.
The paper tackles rain removal in videos with complex dynamic scenes and heavy rainfall by proposing a derain algorithm that uses superpixel segmentation for robust content alignment and a CNN for detail restoration, achieving up to a 5 dB PSNR advantage over state-of-the-art methods.
Rain removal is important for improving the robustness of outdoor vision based systems. Current rain removal methods show limitations either for complex dynamic scenes shot from fast moving cameras, or under torrential rain fall with opaque occlusions. We propose a novel derain algorithm, which applies superpixel (SP) segmentation to decompose the scene into depth consistent units. Alignment of scene contents are done at the SP level, which proves to be robust towards rain occlusion and fast camera motion. Two alignment output tensors, i.e., optimal temporal match tensor and sorted spatial-temporal match tensor, provide informative clues for rain streak location and occluded background contents to generate an intermediate derain output. These tensors will be subsequently prepared as input features for a convolutional neural network to restore high frequency details to the intermediate output for compensation of mis-alignment blur. Extensive evaluations show that up to 5 dB reconstruction PSNR advantage is achieved over state-of-the-art methods. Visual inspection shows that much cleaner rain removal is achieved especially for highly dynamic scenes with heavy and opaque rainfall from a fast moving camera.