SC-Captioner: Improving Image Captioning with Self-Correction by Reinforcement Learning
This addresses the challenge of generating accurate and refined image captions for applications in computer vision and natural language processing, representing an incremental improvement with a novel method for a known bottleneck.
The paper tackles the problem of improving image captioning by introducing SC-Captioner, a reinforcement learning framework that enables self-correction, resulting in significantly better captions that outperform direct preference optimization training.
We propose SC-Captioner, a reinforcement learning framework that enables the self-correcting capability of image caption models. Our crucial technique lies in the design of the reward function to incentivize accurate caption corrections. Specifically, the predicted and reference captions are decomposed into object, attribute, and relation sets using scene-graph parsing algorithms. We calculate the set difference between sets of initial and self-corrected captions to identify added and removed elements. These elements are matched against the reference sets to calculate correctness bonuses for accurate refinements and mistake punishments for wrong additions and removals, thereby forming the final reward. For image caption quality assessment, we propose a set of metrics refined from CAPTURE that alleviate its incomplete precision evaluation and inefficient relation matching problems. Furthermore, we collect a fine-grained annotated image caption dataset, RefinedCaps, consisting of 6.5K diverse images from COCO dataset. Experiments show that applying SC-Captioner on large visual-language models can generate better image captions across various scenarios, significantly outperforming the direct preference optimization training strategy.