What Drives Compositional Generalization in Visual Generative Models?
This work addresses the problem of understanding and enhancing compositional generalization for researchers and practitioners in visual generative AI, though it is incremental as it builds on existing models like MaskGIT.
The study systematically investigates how design choices affect compositional generalization in visual generative models, identifying key factors like discrete vs. continuous training objectives and conditioning information, and shows that adding a continuous JEPA-based objective to MaskGIT improves performance.
Compositional generalization, the ability to generate novel combinations of known concepts, is a key ingredient for visual generative models. Yet, not all mechanisms that enable or inhibit it are fully understood. In this work, we conduct a systematic study of how various design choices influence compositional generalization in image and video generation in a positive or negative way. Through controlled experiments, we identify two key factors: (i) whether the training objective operates on a discrete or continuous distribution, and (ii) to what extent conditioning provides information about the constituent concepts during training. Building on these insights, we show that relaxing the MaskGIT discrete loss with an auxiliary continuous JEPA-based objective can improve compositional performance in discrete models like MaskGIT.