Optimising the Input Image to Improve Visual Relationship Detection
This work addresses a domain-specific bottleneck in computer vision for researchers, but it is incremental as it focuses on preprocessing improvements within an existing framework.
The paper tackled the problem of improving visual relationship detection by optimizing input image preprocessing, finding that the Union-WB-and-B method significantly outperforms the widely used Union method, with concrete gains in recall metrics.
Visual Relationship Detection is defined as, given an image composed of a subject and an object, the correct relation is predicted. To improve the visual part of this difficult problem, ten preprocessing methods were tested to determine whether the widely used Union method yields the optimal results. Therefore, focusing solely on predicate prediction, no object detection and linguistic knowledge were used to prevent them from affecting the comparison results. Once fine-tuned, the Visual Geometry Group models were evaluated using Recall@1, per-predicate recall, activation maximisations, class activation maps, and error analysis. From this research it was found that using preprocessing methods such as the Union-Without-Background-and-with-Binary-mask (Union-WB-and-B) method yields significantly better results than the widely used Union method since, as designed, it enables the Convolutional Neural Network to also identify the subject and object in the convolutional layers instead of solely in the fully-connected layers.