X-Fusion: Introducing New Modality to Frozen Large Language Models
This work addresses the challenge of efficiently integrating new modalities into frozen LLMs for multimodal applications, though it appears incremental as it builds on existing dual-tower designs.
The authors tackled the problem of extending pretrained Large Language Models (LLMs) to multimodal tasks without compromising their language capabilities, achieving consistent performance improvements over alternative architectures on image-to-text and text-to-image tasks.
We propose X-Fusion, a framework that extends pretrained Large Language Models (LLMs) for multimodal tasks while preserving their language capabilities. X-Fusion employs a dual-tower design with modality-specific weights, keeping the LLM's parameters frozen while integrating vision-specific information for both understanding and generation. Our experiments demonstrate that X-Fusion consistently outperforms alternative architectures on both image-to-text and text-to-image tasks. We find that incorporating understanding-focused data improves generation quality, reducing image data noise enhances overall performance, and feature alignment accelerates convergence for smaller models but has minimal impact on larger ones. Our findings provide valuable insights into building efficient unified multimodal models.