Implementing Adaptations for Vision AutoRegressive Model
This work addresses the underexplored area of adaptations for VAR models, which is incremental as it implements existing strategies rather than proposing new ones.
The paper tackles the problem of adapting Vision AutoRegressive models (VAR) for downstream tasks like medical data generation, finding that VAR outperforms Diffusion Models for non-differentially private adaptations but suffers in private settings.
Vision AutoRegressive model (VAR) was recently introduced as an alternative to Diffusion Models (DMs) in image generation domain. In this work we focus on its adaptations, which aim to fine-tune pre-trained models to perform specific downstream tasks, like medical data generation. While for DMs there exist many techniques, adaptations for VAR remain underexplored. Similarly, differentially private (DP) adaptations-ones that aim to preserve privacy of the adaptation data-have been extensively studied for DMs, while VAR lacks such solutions. In our work, we implement and benchmark many strategies for VAR, and compare them to state-of-the-art DM adaptation strategies. We observe that VAR outperforms DMs for non-DP adaptations, however, the performance of DP suffers, which necessitates further research in private adaptations for VAR. Code is available at https://github.com/sprintml/finetuning_var_dp.