Reinforcing an Image Caption Generator Using Off-Line Human Feedback
This addresses the challenge of leveraging limited human feedback to enhance caption quality for AI applications, though it is incremental as it builds on existing reinforcement learning methods.
The paper tackled the problem of improving image captioning models by using instance-level human ratings as rewards in an off-policy reinforcement learning setting, resulting in the model learning to generalize human judgments to unseen images as validated by different human evaluators.
Human ratings are currently the most accurate way to assess the quality of an image captioning model, yet most often the only used outcome of an expensive human rating evaluation is a few overall statistics over the evaluation dataset. In this paper, we show that the signal from instance-level human caption ratings can be leveraged to improve captioning models, even when the amount of caption ratings is several orders of magnitude less than the caption training data. We employ a policy gradient method to maximize the human ratings as rewards in an off-policy reinforcement learning setting, where policy gradients are estimated by samples from a distribution that focuses on the captions in a caption ratings dataset. Our empirical evidence indicates that the proposed method learns to generalize the human raters' judgments to a previously unseen set of images, as judged by a different set of human judges, and additionally on a different, multi-dimensional side-by-side human evaluation procedure.