Decoupled Box Proposal and Featurization with Ultrafine-Grained Semantic Labels Improve Image Captioning and Visual Question Answering
This addresses the problem of costly annotation for vision-language tasks, offering an incremental improvement for researchers and practitioners in AI.
The paper tackles the high annotation cost of object detection models like Faster R-CNN by decoupling box proposal and featurization, enabling the use of previously unavailable labeled data; this approach improves image captioning and visual question answering models on public benchmarks.
Object detection plays an important role in current solutions to vision and language tasks like image captioning and visual question answering. However, popular models like Faster R-CNN rely on a costly process of annotating ground-truths for both the bounding boxes and their corresponding semantic labels, making it less amenable as a primitive task for transfer learning. In this paper, we examine the effect of decoupling box proposal and featurization for down-stream tasks. The key insight is that this allows us to leverage a large amount of labeled annotations that were previously unavailable for standard object detection benchmarks. Empirically, we demonstrate that this leads to effective transfer learning and improved image captioning and visual question answering models, as measured on publicly available benchmarks.