RelVAE: Generative Pretraining for few-shot Visual Relationship Detection
This addresses the data scarcity issue in visual relationship detection for computer vision applications, though it appears incremental as it builds on existing few-shot learning approaches.
The paper tackles the problem of few-shot visual relationship detection by introducing RelVAE, a generative pretraining method that captures semantic, visual, and spatial information of relations without requiring annotated data, and shows it outperforms baselines on VG200 and VRD datasets.
Visual relations are complex, multimodal concepts that play an important role in the way humans perceive the world. As a result of their complexity, high-quality, diverse and large scale datasets for visual relations are still absent. In an attempt to overcome this data barrier, we choose to focus on the problem of few-shot Visual Relationship Detection (VRD), a setting that has been so far neglected by the community. In this work we present the first pretraining method for few-shot predicate classification that does not require any annotated relations. We achieve this by introducing a generative model that is able to capture the variation of semantic, visual and spatial information of relations inside a latent space and later exploiting its representations in order to achieve efficient few-shot classification. We construct few-shot training splits and show quantitative experiments on VG200 and VRD datasets where our model outperforms the baselines. Lastly we attempt to interpret the decisions of the model by conducting various qualitative experiments.