Survey: Transformer-based Models in Data Modality Conversion
It provides a comprehensive overview for researchers and practitioners in AI, but it is incremental as it synthesizes existing literature without new methods or results.
This survey addresses the lack of a systematic review of transformer-based models for data modality conversion, covering text, vision, and speech to highlight their versatility in AI content generation.
Transformers have made significant strides across various artificial intelligence domains, including natural language processing, computer vision, and audio processing. This success has naturally garnered considerable interest from both academic and industry researchers. Consequently, numerous Transformer variants (often referred to as X-formers) have been developed for these fields. However, a thorough and systematic review of these modality-specific conversions remains lacking. Modality Conversion involves the transformation of data from one form of representation to another, mimicking the way humans integrate and interpret sensory information. This paper provides a comprehensive review of transformer-based models applied to the primary modalities of text, vision, and speech, discussing their architectures, conversion methodologies, and applications. By synthesizing the literature on modality conversion, this survey aims to underline the versatility and scalability of transformers in advancing AI-driven content generation and understanding.