Mural: Transferring LLM knowledge to image generation via Mixture-of-Transformers
For researchers in multimodal learning, this work shows that frozen LLM knowledge can be effectively leveraged in standard text-to-image training, enabling emergent capabilities without additional supervision.
The paper investigates transferring knowledge from a frozen LLM to text-to-image generation using a Mixture-of-Transformers architecture, achieving strong benchmark scores (0.85 GenEval, 86.75 DPG-Bench, 0.66 WISE) and discovering emergent cross-lingual, compositional, and world-knowledge-guided image generation without multimodal training data.
Leveraging capabilities of large language models (LLMs) in text-to-image (T2I) synthesis is an important research direction. In this work we investigate whether the knowledge of a frozen LLM can be effectively utilized in T2I generation when trained exclusively on standard text-image pairs. We integrate a frozen, reasoning-capable LLM with a diffusion-based image generator via shared attention within the Mixture-of-Transformers (MoT) architecture. Our experiments span two critical questions: (1) what degree of the LLM's intrinsic knowledge remains accessible during T2I training, and (2) what novel capabilities emerge in the resulting system. Across established benchmarks, our models achieve strong performance among unified understanding-generation systems: 0.85 on GenEval, 86.75 on DPG-Bench, and 0.66 on WISE with inference-time reasoning, using only text-image data. Remarkably, we uncover emergent behaviors absent from training data, including cross-lingual image generation, color-guided composition, emoji / ASCII scene construction, and generation directed by world knowledge. These results demonstrate that pretrained LLM knowledge can guide image synthesis under standard text-to-image training paradigms, without interleaved multimodal signals or explicit reasoning supervision. Our findings open new avenues for harnessing frozen model capabilities in resource-constrained multimodal learning.