RS-Net: Context-Aware Relation Scoring for Dynamic Scene Graph Generation
This work addresses a specific bottleneck in video understanding for computer vision researchers, offering an incremental improvement by enhancing relation prediction in dynamic scenes.
The paper tackled the problem of Dynamic Scene Graph Generation in videos, where existing methods struggle to identify meaningful relations due to lack of guidance for non-related pairs, and proposed RS-Net, a modular framework that scores contextual importance of object pairs using spatial and temporal context, resulting in improved Recall and Precision on the Action Genome dataset.
Dynamic Scene Graph Generation (DSGG) models how object relations evolve over time in videos. However, existing methods are trained only on annotated object pairs and lack guidance for non-related pairs, making it difficult to identify meaningful relations during inference. In this paper, we propose Relation Scoring Network (RS-Net), a modular framework that scores the contextual importance of object pairs using both spatial interactions and long-range temporal context. RS-Net consists of a spatial context encoder with learnable context tokens and a temporal encoder that aggregates video-level information. The resulting relation scores are integrated into a unified triplet scoring mechanism to enhance relation prediction. RS-Net can be easily integrated into existing DSGG models without architectural changes. Experiments on the Action Genome dataset show that RS-Net consistently improves both Recall and Precision across diverse baselines, with notable gains in mean Recall, highlighting its ability to address the long-tailed distribution of relations. Despite the increased number of parameters, RS-Net maintains competitive efficiency, achieving superior performance over state-of-the-art methods.