i-Code V2: An Autoregressive Generation Framework over Vision, Language, and Speech Data
This addresses the problem of multimodal AI integration for researchers and developers by enabling generative capabilities across diverse data types, though it is incremental as it builds on existing encoders.
The authors tackled the lack of generative abilities in multimodal models by proposing i-Code V2, the first model that generates natural language from any combination of vision, language, and speech data, matching or outperforming state-of-the-art baselines on 7 multimodal tasks.
The convergence of text, visual, and audio data is a key step towards human-like artificial intelligence, however the current Vision-Language-Speech landscape is dominated by encoder-only models which lack generative abilities. We propose closing this gap with i-Code V2, the first model capable of generating natural language from any combination of Vision, Language, and Speech data. i-Code V2 is an integrative system that leverages state-of-the-art single-modality encoders, combining their outputs with a new modality-fusing encoder in order to flexibly project combinations of modalities into a shared representational space. Next, language tokens are generated from these representations via an autoregressive decoder. The whole framework is pretrained end-to-end on a large collection of dual- and single-modality datasets using a novel text completion objective that can be generalized across arbitrary combinations of modalities. i-Code V2 matches or outperforms state-of-the-art single- and dual-modality baselines on 7 multimodal tasks, demonstrating the power of generative multimodal pretraining across a diversity of tasks and signals.