Improving Image Captioning with Control Signal of Sentence Quality
This work addresses the issue of inconsistent caption quality in image captioning datasets, offering an incremental improvement for AI vision-language tasks.
The paper tackles the problem of varying quality in image captions by introducing a control signal for sentence quality as an additional input to captioning models, enabling them to handle different quality levels differently, and results show that models using the highest quality level outperform baselines on accuracy metrics in MSCOCO experiments.
In the dataset of image captioning, each image is aligned with several descriptions. Despite the fact that the quality of these descriptions varies, existing captioning models treat them equally in the training process. In this paper, we propose a new control signal of sentence quality, which is taken as an additional input to the captioning model. By integrating the control signal information, captioning models are aware of the quality level of the target sentences and handle them differently. Moreover, we propose a novel reinforcement training method specially designed for the control signal of sentence quality: Quality-oriented Self-Annotated Training (Q-SAT). Extensive experiments on MSCOCO dataset show that without extra information from ground truth captions, models controlled by the highest quality level outperform baseline models on accuracy-based evaluation metrics, which validates the effectiveness of our proposed methods.