UniCode$^2$: Cascaded Large-scale Codebooks for Unified Multimodal Understanding and Generation
This addresses the challenge of improving multimodal understanding and generation for AI systems, though it is incremental as it builds on existing codebook-based methods.
The paper tackled the problem of scaling visual codebooks for unified multimodal models, which often suffer from low token utilization and unstable training, by proposing UniCode^2, a cascaded framework that achieved strong performance across diverse benchmarks with a 500K-entry codebook.
Unified multimodal large language models (MLLMs) have shown promise in jointly advancing multimodal understanding and generation, with visual codebooks discretizing images into tokens for autoregressive modeling. Existing codebook-based methods either rely on small vocabularies (~16K entries) that lack fine-grained semantics or naively scale up, resulting in low token utilization and unstable training. We propose UniCode$^2$, a cascaded codebook framework enabling large-scale, semantically aligned, and stable visual tokenization. By clustering millions of SigLIP sequence embeddings, we build a 500K-entry codebook that preserves vision-language alignment while expanding capacity. Stability is ensured via a cascaded design: a frozen codebook anchors the embedding space, and a trainable codebook refines task-specific semantics. This decoupling promotes high utilization and robust learning. Moreover, the alignment of our visual tokens with textual semantics enables seamless integration with pretrained diffusion decoders, supporting high-quality visual synthesis with minimal adaptation. UniCode^2 delivers strong performance across diverse benchmarks, demonstrating the viability of scaling visual token spaces without sacrificing stability, semantics, or modularity.