Frequency-guided Multi-level Reasoning for Scene Graph Generation in Video
For researchers in video understanding, this work addresses the long-tail problem in scene graph generation, offering a mechanism-level solution that improves tail-class recall.
The paper tackles long-tail distribution in Video Scene Graph Generation by proposing FReMuRe, which uses relation-specific branches and frequency-aware dual-branch predicate embedding to improve tail-class recall. On Action Genome, it achieves significant gains in long-tail relationship recall and overall robustness.
Video Scene Graph Generation aims to obtain structured semantic representations of objects and their relationships in videos for high-level understanding. However, existing methods still have limitations in handling long-tail distributions. This paper proposes the Frequency-guided Relational Multi-level Reasoning (FReMuRe) model, which enhances the modeling ability of long-tail relationships from a mechanism perspective. We introduce relation-specific branches to deal gradient conflicts, yielding more balanced and tail-aware learning. And we design a frequency-aware dual-branch predicate embedding network to model high-frequency and low-frequency relationships separately and improve the recall rate of tail classes through gated fusion. Meanwhile, we propose two types of interchangeable relation classification heads: Bayesian Head for uncertainty estimation and new Gaussian Mixture Model Head to enhance intra-class diversity. Experimental results show that FReMuRe significantly improves the recall rate of long-tail relationships and overall reasoning robustness on the Action Genome dataset.