MEPA: Multi-Scale Representation Alignment for Visual Autoregressive Modeling with Mixture of Experts
This work addresses optimization conflicts and error propagation in multi-scale autoregressive image generation, offering a more efficient and higher-quality alternative for generative modeling.
MEPA introduces a scale-aware token-routed Mixture of Experts (MoE) architecture and residual feature aggregation to improve multi-scale representation learning in Visual AutoRegressive (VAR) models, achieving superior FID on ImageNet 256×256 with half the training epochs and smaller parameter budget compared to the dense baseline.
Visual AutoRegressive modeling (VAR) has pioneered a coarse-to-fine multi-scale autoregressive generative paradigm, demonstrating strong capabilities in image generation. However, VAR still suffers from inherent deficiencies in multi-scale representation learning. Specifically, lower scales primarily capture global semantics, while higher scales focus on fine-grained details. Employing a shared architecture across scales induces optimization conflicts. Moreover, due to the causal autoregressive process, inaccurate semantics at early scales can propagate and significantly degrade the final output. To address these issues, we introduce a scale-aware token-routed Mixture of Experts (MoE) architecture, allowing scale-adaptive expert selection, thereby facilitating decoupled representation learning across scales. In addition, we enhance semantic modeling at early scales by incorporating external self-supervised features. Unlike naive alignment, we analyse and design a residual feature aggregation scheme tailored to the VAR paradigm. Extensive experiments show that our method significantly improves both training efficiency and generation quality. On the ImageNet 256*256 benchmark, our model achieves a superior FID compared to the dense baseline while requiring only half of the default training epochs and a smaller parameter budget, with a merely marginal increase in training cost. Moreover, the performance gap further widens with larger training epochs.