SPAE: Semantic Pyramid AutoEncoder for Multimodal Generation with Frozen LLMs
This addresses the challenge of integrating visual content with frozen LLMs for multimodal AI applications, representing a novel advancement rather than an incremental improvement.
The paper tackles the problem of enabling frozen large language models (LLMs) to handle non-linguistic modalities like images by introducing SPAE, which converts pixels into interpretable lexical tokens, allowing the LLMs to perform multimodal understanding and generation tasks with over 25% improvement in image understanding performance.
In this work, we introduce Semantic Pyramid AutoEncoder (SPAE) for enabling frozen LLMs to perform both understanding and generation tasks involving non-linguistic modalities such as images or videos. SPAE converts between raw pixels and interpretable lexical tokens (or words) extracted from the LLM's vocabulary. The resulting tokens capture both the semantic meaning and the fine-grained details needed for visual reconstruction, effectively translating the visual content into a language comprehensible to the LLM, and empowering it to perform a wide array of multimodal tasks. Our approach is validated through in-context learning experiments with frozen PaLM 2 and GPT 3.5 on a diverse set of image understanding and generation tasks. Our method marks the first successful attempt to enable a frozen LLM to generate image content while surpassing state-of-the-art performance in image understanding tasks, under the same setting, by over 25%.