Learning to Select: A Fully Attentive Approach for Novel Object Captioning
This addresses the problem of real-life image captioning where training data is limited, offering an incremental advance in handling unseen objects.
The paper tackles the challenge of generating captions for images containing objects not seen during training, known as novel object captioning (NOC), by proposing a fully-attentive method that selects relevant objects and constrains language generation, resulting in improvements in adaptability and caption quality on the held-out COCO dataset.
Image captioning models have lately shown impressive results when applied to standard datasets. Switching to real-life scenarios, however, constitutes a challenge due to the larger variety of visual concepts which are not covered in existing training sets. For this reason, novel object captioning (NOC) has recently emerged as a paradigm to test captioning models on objects which are unseen during the training phase. In this paper, we present a novel approach for NOC that learns to select the most relevant objects of an image, regardless of their adherence to the training set, and to constrain the generative process of a language model accordingly. Our architecture is fully-attentive and end-to-end trainable, also when incorporating constraints. We perform experiments on the held-out COCO dataset, where we demonstrate improvements over the state of the art, both in terms of adaptability to novel objects and caption quality.