Semi-Autoregressive Transformer for Image Captioning
This work addresses inference latency issues in image captioning for applications requiring faster generation, though it is incremental as it builds on existing non-autoregressive and autoregressive approaches.
The paper tackles the trade-off between speed and quality in image captioning by introducing a semi-autoregressive model (SATIC) that generates words in parallel locally while maintaining autoregressive properties globally, achieving competitive results on the MSCOCO benchmark.
Current state-of-the-art image captioning models adopt autoregressive decoders, \ie they generate each word by conditioning on previously generated words, which leads to heavy latency during inference. To tackle this issue, non-autoregressive image captioning models have recently been proposed to significantly accelerate the speed of inference by generating all words in parallel. However, these non-autoregressive models inevitably suffer from large generation quality degradation since they remove words dependence excessively. To make a better trade-off between speed and quality, we introduce a semi-autoregressive model for image captioning~(dubbed as SATIC), which keeps the autoregressive property in global but generates words parallelly in local . Based on Transformer, there are only a few modifications needed to implement SATIC. Experimental results on the MSCOCO image captioning benchmark show that SATIC can achieve a good trade-off without bells and whistles. Code is available at {\color{magenta}\url{https://github.com/YuanEZhou/satic}}.