4DSegStreamer: Streaming 4D Panoptic Segmentation via Dual Threads
This addresses the need for fine-grained, real-time perception in highly dynamic scenarios, representing an incremental improvement by adapting existing segmentation methods for streaming settings.
The paper tackles real-time 4D panoptic segmentation for dynamic environments like autonomous driving by introducing 4DSegStreamer, a dual-thread framework that integrates with existing methods and shows superior robustness under high FPS conditions.
4D panoptic segmentation in a streaming setting is critical for highly dynamic environments, such as evacuating dense crowds and autonomous driving in complex scenarios, where real-time, fine-grained perception within a constrained time budget is essential. In this paper, we introduce 4DSegStreamer, a novel framework that employs a Dual-Thread System to efficiently process streaming frames. The framework is general and can be seamlessly integrated into existing 3D and 4D segmentation methods to enable real-time capability. It also demonstrates superior robustness compared to existing streaming perception approaches, particularly under high FPS conditions. The system consists of a predictive thread and an inference thread. The predictive thread leverages historical motion and geometric information to extract features and forecast future dynamics. The inference thread ensures timely prediction for incoming frames by aligning with the latest memory and compensating for ego-motion and dynamic object movements. We evaluate 4DSegStreamer on the indoor HOI4D dataset and the outdoor SemanticKITTI and nuScenes datasets. Comprehensive experiments demonstrate the effectiveness of our approach, particularly in accurately predicting dynamic objects in complex scenes.