Value of Temporal Dynamics Information in Driving Scene Segmentation
This work addresses the problem of improving semantic segmentation in dynamic scenes for autonomous driving applications, but it is incremental as it builds on existing video segmentation approaches.
The paper investigates how much temporal dynamics information contributes to driving scene segmentation beyond single frames, by combining appearance and temporal representations in a joint-learning framework, and introduces the MIT Driving Scene Segmentation dataset with 5,000 densely annotated video frames.
Semantic scene segmentation has primarily been addressed by forming representations of single images both with supervised and unsupervised methods. The problem of semantic segmentation in dynamic scenes has begun to recently receive attention with video object segmentation approaches. What is not known is how much extra information the temporal dynamics of the visual scene carries that is complimentary to the information available in the individual frames of the video. There is evidence that the human visual system can effectively perceive the scene from temporal dynamics information of the scene's changing visual characteristics without relying on the visual characteristics of individual snapshots themselves. Our work takes steps to explore whether machine perception can exhibit similar properties by combining appearance-based representations and temporal dynamics representations in a joint-learning problem that reveals the contribution of each toward successful dynamic scene segmentation. Additionally, we provide the MIT Driving Scene Segmentation dataset, which is a large-scale full driving scene segmentation dataset, densely annotated for every pixel and every one of 5,000 video frames. This dataset is intended to help further the exploration of the value of temporal dynamics information for semantic segmentation in video.