Tensor Composition Net for Visual Relationship Prediction
This work addresses the problem of visual relationship prediction for computer vision systems, offering a method to handle large label spaces and incomplete annotations, which is an incremental improvement over existing methods.
This paper introduces the Tensor Composition Net (TCN) to address the challenge of Visual Relationship Prediction (VRP) in images, which suffers from a large label-space and incomplete annotation. The TCN leverages the low-rank property of the visual relationship tensor to make structured predictions, enabling the prediction of unseen visual relationships and facilitating relation-based image retrieval.
We present a novel Tensor Composition Net (TCN) to predict visual relationships in images. Visual Relationship Prediction (VRP) provides a more challenging test of image understanding than conventional image tagging and is difficult to learn due to a large label-space and incomplete annotation. The key idea of our TCN is to exploit the low-rank property of the visual relationship tensor, so as to leverage correlations within and across objects and relations and make a structured prediction of all visual relationships in an image. To show the effectiveness of our model, we first empirically compare our model with Multi-Label Image Classification (MLIC) methods, eXtreme Multi-label Classification (XMC) methods, and VRD methods. We then show that thanks to our tensor (de)composition layer, our model can predict visual relationships which have not been seen in the training dataset. We finally show our TCN's image-level visual relationship prediction provides a simple and efficient mechanism for relation-based image-retrieval even compared with VRD methods.