Object-Centric Representation Learning with Generative Spatial-Temporal Factorization
This work addresses the challenge of unsupervised object-centric representation learning for dynamic scenes, which is crucial for structural understanding in AI and robotics, representing a novel extension beyond static or single-view assumptions.
The paper tackles the problem of learning object-centric scene representations from dynamic scenes without supervision, addressing limitations of existing methods that assume stationary observers or static scenes. The proposed DyMON method successfully factorizes observer motions and object dynamics from multi-view sequences, enabling rendering from arbitrary viewpoints and times, and supports independent querying of individual objects across space and time.
Learning object-centric scene representations is essential for attaining structural understanding and abstraction of complex scenes. Yet, as current approaches for unsupervised object-centric representation learning are built upon either a stationary observer assumption or a static scene assumption, they often: i) suffer single-view spatial ambiguities, or ii) infer incorrectly or inaccurately object representations from dynamic scenes. To address this, we propose Dynamics-aware Multi-Object Network (DyMON), a method that broadens the scope of multi-view object-centric representation learning to dynamic scenes. We train DyMON on multi-view-dynamic-scene data and show that DyMON learns -- without supervision -- to factorize the entangled effects of observer motions and scene object dynamics from a sequence of observations, and constructs scene object spatial representations suitable for rendering at arbitrary times (querying across time) and from arbitrary viewpoints (querying across space). We also show that the factorized scene representations (w.r.t. objects) support querying about a single object by space and time independently.