Multiscale Video Transformers for Class Agnostic Segmentation in Autonomous Driving
This addresses safety challenges in autonomous driving by enabling robust detection of unknown objects, though it is incremental as it builds on existing transformer and segmentation methods.
The paper tackles the problem of detecting unknown objects in autonomous driving by developing multiscale video transformers that use motion cues for class-agnostic segmentation, achieving consistent performance improvements over baselines on datasets like DAVIS'16, KITTI, and Cityscapes while being efficient in GPU memory and run-time.
Ensuring safety in autonomous driving is a complex challenge requiring handling unknown objects and unforeseen driving scenarios. We develop multiscale video transformers capable of detecting unknown objects using only motion cues. Video semantic and panoptic segmentation often relies on known classes seen during training, overlooking novel categories. Recent visual grounding with large language models is computationally expensive, especially for pixel-level output. We propose an efficient video transformer trained end-to-end for class-agnostic segmentation without optical flow. Our method uses multi-stage multiscale query-memory decoding and a scale-specific random drop-token to ensure efficiency and accuracy, maintaining detailed spatiotemporal features with a shared, learnable memory module. Unlike conventional decoders that compress features, our memory-centric design preserves high-resolution information at multiple scales. We evaluate on DAVIS'16, KITTI, and Cityscapes. Our method consistently outperforms multiscale baselines while being efficient in GPU memory and run-time, demonstrating a promising direction for real-time, robust dense prediction in safety-critical robotics.