MetaMorph: Multimodal Understanding and Generation via Instruction Tuning
This work addresses the challenge of unifying multimodal AI capabilities for applications requiring both image and text processing, though it is incremental as it builds on existing instruction tuning methods.
The paper tackles the problem of enabling large language models (LLMs) to handle both visual understanding and generation by proposing Visual-Predictive Instruction Tuning (VPiT), which allows an LLM to predict text and visual tokens from multimodal inputs, resulting in competitive performance on both tasks with efficient adaptation using a small amount of generation data.
In this work, we propose Visual-Predictive Instruction Tuning (VPiT) - a simple and effective extension to visual instruction tuning that enables a pretrained LLM to quickly morph into an unified autoregressive model capable of generating both text and visual tokens. VPiT teaches an LLM to predict discrete text tokens and continuous visual tokens from any input sequence of image and text data curated in an instruction-following format. Our empirical investigation reveals several intriguing properties of VPiT: (1) visual generation ability emerges as a natural byproduct of improved visual understanding, and can be unlocked efficiently with a small amount of generation data; (2) while we find understanding and generation to be mutually beneficial, understanding data contributes to both capabilities more effectively than generation data. Building upon these findings, we train our MetaMorph model and achieve competitive performance on both visual understanding and generation. In visual generation, MetaMorph can leverage the world knowledge and reasoning abilities gained from LLM pretraining, and overcome common failure modes exhibited by other generation models. Our results suggest that LLMs may have strong "prior" vision capabilities that can be efficiently adapted to both visual understanding and generation with a relatively simple instruction tuning process.