DSFlash: Comprehensive Panoptic Scene Graph Generation in Realtime
This enables real-time, comprehensive scene graph generation for researchers and practitioners with limited computational resources, though it is incremental in improving efficiency.
The paper tackles the problem of slow and resource-intensive scene graph generation for practical deployment on edge devices by introducing DSFlash, a low-latency model that processes video at 56 FPS on an RTX 3090 GPU while matching state-of-the-art performance and requiring less than 24 hours to train on a GTX 1080 GPU.
Scene Graph Generation (SGG) aims to extract a detailed graph structure from an image, a representation that holds significant promise as a robust intermediate step for complex downstream tasks like reasoning for embodied agents. However, practical deployment in real-world applications - especially on resource constrained edge devices - requires speed and resource efficiency, challenges that have received limited attention in existing research. To bridge this gap, we introduce DSFlash, a low-latency model for panoptic scene graph generation designed to overcome these limitations. DSFlash can process a video stream at 56 frames per second on a standard RTX 3090 GPU, without compromising performance against existing state-of-the-art methods. Crucially, unlike prior approaches that often restrict themselves to salient relationships, DSFlash computes comprehensive scene graphs, offering richer contextual information while maintaining its superior latency. Furthermore, DSFlash is light on resources, requiring less than 24 hours to train on a single, nine-year-old GTX 1080 GPU. This accessibility makes DSFlash particularly well-suited for researchers and practitioners operating with limited computational resources, empowering them to adapt and fine-tune SGG models for specialized applications.