M3D-GAN: Multi-Modal Multi-Domain Translation with Universal Attention
This addresses the need for versatile translation models across multiple modalities and domains, though it appears incremental as it builds on existing GAN frameworks.
The paper tackles the problem of cross-modal and cross-domain translation by proposing M3D-GAN, a unified model that translates across modalities like text, image, and speech, achieving state-of-the-art performance on some benchmark tasks.
Generative adversarial networks have led to significant advances in cross-modal/domain translation. However, typically these networks are designed for a specific task (e.g., dialogue generation or image synthesis, but not both). We present a unified model, M3D-GAN, that can translate across a wide range of modalities (e.g., text, image, and speech) and domains (e.g., attributes in images or emotions in speech). Our model consists of modality subnets that convert data from different modalities into unified representations, and a unified computing body where data from different modalities share the same network architecture. We introduce a universal attention module that is jointly trained with the whole network and learns to encode a large range of domain information into a highly structured latent space. We use this to control synthesis in novel ways, such as producing diverse realistic pictures from a sketch or varying the emotion of synthesized speech. We evaluate our approach on extensive benchmark tasks, including image-to-image, text-to-image, image captioning, text-to-speech, speech recognition, and machine translation. Our results show state-of-the-art performance on some of the tasks.