Putting Humans in the Image Captioning Loop
This work addresses the need for customizable image captioning models, but it is incremental as it builds on existing pre-trained models and feedback mechanisms.
The paper tackles the problem of limited data in image captioning by integrating human feedback into the training process, using a base model pre-trained on MS COCO and step-wise updates with memory replay to adapt to user-specific data.
Image Captioning (IC) models can highly benefit from human feedback in the training process, especially in cases where data is limited. We present work-in-progress on adapting an IC system to integrate human feedback, with the goal to make it easily adaptable to user-specific data. Our approach builds on a base IC model pre-trained on the MS COCO dataset, which generates captions for unseen images. The user will then be able to offer feedback on the image and the generated/predicted caption, which will be augmented to create additional training instances for the adaptation of the model. The additional instances are integrated into the model using step-wise updates, and a sparse memory replay component is used to avoid catastrophic forgetting. We hope that this approach, while leading to improved results, will also result in customizable IC models.