Semantic-Aware Prefix Learning for Token-Efficient Image Generation
This work addresses the issue of semantic alignment in latent image generation for researchers and practitioners, offering incremental improvements over existing methods.
The paper tackled the problem of weakly grounded semantic representations in visual tokenizers by proposing SMAP, a semantic-aware prefix tokenizer that injects class-level semantic conditions and uses a tail token dropping strategy, resulting in improved reconstruction quality and strong downstream generation performance on ImageNet under compact token budgets.
Visual tokenizers play a central role in latent image generation by bridging high-dimensional images and tractable generative modeling. However, most existing tokenizers are still trained with reconstruction-dominated objectives, which often yield latent representations that are only weakly grounded in high-level semantics. Recent approaches improve semantic alignment, but typically treat semantic signals as auxiliary regularization rather than making them functionally necessary for representation learning. We propose SMAP, a SeMantic-Aware Prefix tokenizer that injects class-level semantic conditions into a query-based 1D tokenization framework. To make semantics indispensable during training, SMAP introduces a tail token dropping strategy, which forces semantic conditions and early latent prefixes to bear increasing responsibility under progressively reduced token budgets. To verify that the resulting latent space is useful for generation rather than reconstruction alone, we further introduce CARD, a hybrid Causal AutoRegressive--Diffusion generator. Extensive experiments on ImageNet show that SMAP consistently improves reconstruction quality across discrete and continuous tokenization settings, and that its semantically grounded latent space yields strong downstream generation performance under compact token budgets.