Knowledge-augmented Few-shot Visual Relation Detection
This addresses the challenge of detecting visual relationships with limited training data, which is crucial for image understanding applications, though it appears incremental by building on existing few-shot learning approaches.
The paper tackles the problem of few-shot visual relation detection by introducing a knowledge-augmented framework that leverages textual and visual knowledge to improve generalization, achieving state-of-the-art performance with a large improvement on benchmarks from the Visual Genome dataset.
Visual Relation Detection (VRD) aims to detect relationships between objects for image understanding. Most existing VRD methods rely on thousands of training samples of each relationship to achieve satisfactory performance. Some recent papers tackle this problem by few-shot learning with elaborately designed pipelines and pre-trained word vectors. However, the performance of existing few-shot VRD models is severely hampered by the poor generalization capability, as they struggle to handle the vast semantic diversity of visual relationships. Nonetheless, humans have the ability to learn new relationships with just few examples based on their knowledge. Inspired by this, we devise a knowledge-augmented, few-shot VRD framework leveraging both textual knowledge and visual relation knowledge to improve the generalization ability of few-shot VRD. The textual knowledge and visual relation knowledge are acquired from a pre-trained language model and an automatically constructed visual relation knowledge graph, respectively. We extensively validate the effectiveness of our framework. Experiments conducted on three benchmarks from the commonly used Visual Genome dataset show that our performance surpasses existing state-of-the-art models with a large improvement.