DiffCap: Exploring Continuous Diffusion on Image Captioning
This addresses the problem of limited decoding diversity in image captioning for researchers and practitioners, though it is incremental as it adapts existing diffusion techniques to a new task.
The authors tackled image captioning by applying continuous diffusion models to generate captions non-autoregressively, achieving comparable results to previous non-autoregressive methods on the COCO dataset with a simpler structure and high diversity in generation.
Current image captioning works usually focus on generating descriptions in an autoregressive manner. However, there are limited works that focus on generating descriptions non-autoregressively, which brings more decoding diversity. Inspired by the success of diffusion models on generating natural-looking images, we propose a novel method DiffCap to apply continuous diffusions on image captioning. Unlike image generation where the output is fixed-size and continuous, image description length varies with discrete tokens. Our method transforms discrete tokens in a natural way and applies continuous diffusion on them to successfully fuse extracted image features for diffusion caption generation. Our experiments on COCO dataset demonstrate that our method uses a much simpler structure to achieve comparable results to the previous non-autoregressive works. Apart from quality, an intriguing property of DiffCap is its high diversity during generation, which is missing from many autoregressive models. We believe our method on fusing multimodal features in diffusion language generation will inspire more researches on multimodal language generation tasks for its simplicity and decoding flexibility.