Are Object-Centric Representations Better At Compositional Generalization?
This work addresses the challenge of compositional generalization for machine learning systems in vision tasks, providing systematic evidence that object-centric representations offer advantages in resource-constrained scenarios, though it is incremental in comparing existing methods.
The paper tackles the problem of compositional generalization in visually rich settings by comparing object-centric and dense vision encoders on a Visual Question Answering benchmark across three controlled visual worlds. The results show that object-centric approaches are superior in harder generalization settings, more sample efficient, and outperform dense encoders when dataset size, diversity, or compute is constrained.
Compositional generalization, the ability to reason about novel combinations of familiar concepts, is fundamental to human cognition and a critical challenge for machine learning. Object-centric (OC) representations, which encode a scene as a set of objects, are often argued to support such generalization, but systematic evidence in visually rich settings is limited. We introduce a Visual Question Answering benchmark across three controlled visual worlds (CLEVRTex, Super-CLEVR, and MOVi-C) to measure how well vision encoders, with and without object-centric biases, generalize to unseen combinations of object properties. To ensure a fair and comprehensive comparison, we carefully account for training data diversity, sample size, representation size, downstream model capacity, and compute. We use DINOv2 and SigLIP2, two widely used vision encoders, as the foundation models and their OC counterparts. Our key findings reveal that (1) OC approaches are superior in harder compositional generalization settings; (2) original dense representations surpass OC only on easier settings and typically require substantially more downstream compute; and (3) OC models are more sample efficient, achieving stronger generalization with fewer images, whereas dense encoders catch up or surpass them only with sufficient data and diversity. Overall, object-centric representations offer stronger compositional generalization when any one of dataset size, training data diversity, or downstream compute is constrained.