Look and Modify: Modification Networks for Image Captioning
This addresses the challenge of refining image captions for applications like accessibility or content generation, though it is incremental as it builds on existing captioning frameworks.
The paper tackles the problem of improving image captions by introducing a framework that learns to modify existing captions rather than generating them from scratch, resulting in better evaluation scores on the COCO dataset.
Attention-based neural encoder-decoder frameworks have been widely used for image captioning. Many of these frameworks deploy their full focus on generating the caption from scratch by relying solely on the image features or the object detection regional features. In this paper, we introduce a novel framework that learns to modify existing captions from a given framework by modeling the residual information, where at each timestep the model learns what to keep, remove or add to the existing caption allowing the model to fully focus on "what to modify" rather than on "what to predict". We evaluate our method on the COCO dataset, trained on top of several image captioning frameworks and show that our model successfully modifies captions yielding better ones with better evaluation scores.