Using Artificial Tokens to Control Languages for Multilingual Image Caption Generation
This work addresses the need for more compact multilingual captioning systems, though it is incremental as it builds on existing neural architectures.
The authors tackled the problem of building a single unified model for multilingual image captioning, proposing a simple technique using artificial tokens to control language, and demonstrated that a typical neural architecture can learn to switch between English and Japanese captions.
Recent work in computer vision has yielded impressive results in automatically describing images with natural language. Most of these systems generate captions in a sin- gle language, requiring multiple language-specific models to build a multilingual captioning system. We propose a very simple technique to build a single unified model across languages, using artificial tokens to control the language, making the captioning system more compact. We evaluate our approach on generating English and Japanese captions, and show that a typical neural captioning architecture is capable of learning a single model that can switch between two different languages.