Knowledge Distillation for Visual Autoregressive Models
For researchers and practitioners working on efficient visual autoregressive generation, this paper provides a tailored distillation framework that addresses unique challenges in the image domain.
This work presents the first systematic study of knowledge distillation for visual autoregressive models, revealing that language-based distillation methods fail due to long decoding horizons and token ambiguity. The proposed VarKD framework selectively applies teacher supervision on student samples, consistently outperforming prior baselines on ImageNet and narrowing the gap to large-scale models.
Autoregressive (AR) image generation models are highly expressive but computationally intensive, motivating effective model compression. Knowledge distillation (KD) is a natural approach for model compression and has been widely studied in language modeling, yet its behavior in visual AR generation remains underexplored. In this work, we present the first systematic study of distillation strategies for AR image models. Our analysis shows that while standard distillation can yield meaningful gains, recent methods developed for language do not directly transfer to images: long decoding horizons and visual token ambiguity make teacher supervision unreliable especially under student-conditioned contexts. To address this, we propose VarKD, a distillation framework for visual autoregressive models that distills on student samples while selectively applying teacher supervision and reducing token-level ambiguity. Experiments on ImageNet across multiple AR backbones show that VarKD consistently outperforms prior distillation baselines, narrowing the gap to large-scale models.