What is not where: the challenge of integrating spatial representations into deep learning architectures
This addresses a key limitation in image captioning for applications requiring precise spatial descriptions, but it is incremental as it critiques existing methods without proposing a new solution.
The paper argues that current deep learning image captioning systems fail to capture fine-grained spatial relationships between objects, focusing only on object presence, and suggests that integrating geometric modeling could improve this.
This paper examines to what degree current deep learning architectures for image caption generation capture spatial language. On the basis of the evaluation of examples of generated captions from the literature we argue that systems capture what objects are in the image data but not where these objects are located: the captions generated by these systems are the output of a language model conditioned on the output of an object detector that cannot capture fine-grained location information. Although language models provide useful knowledge for image captions, we argue that deep learning image captioning architectures should also model geometric relations between objects.