SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decomposition
This addresses the challenge of scene understanding for agents in AI, enabling object representation independent of viewpoint, though it is incremental as it builds on existing unsupervised video decomposition methods.
The paper tackles the problem of inferring compositional scene structure and agent viewpoint from RGB video, presenting SIMONe, an unsupervised variational model that learns to factorize video into view-invariant object latents and time-varying frame latents, demonstrating capabilities in view synthesis and instance segmentation across three procedurally generated datasets.
To help agents reason about scenes in terms of their building blocks, we wish to extract the compositional structure of any given scene (in particular, the configuration and characteristics of objects comprising the scene). This problem is especially difficult when scene structure needs to be inferred while also estimating the agent's location/viewpoint, as the two variables jointly give rise to the agent's observations. We present an unsupervised variational approach to this problem. Leveraging the shared structure that exists across different scenes, our model learns to infer two sets of latent representations from RGB video input alone: a set of "object" latents, corresponding to the time-invariant, object-level contents of the scene, as well as a set of "frame" latents, corresponding to global time-varying elements such as viewpoint. This factorization of latents allows our model, SIMONe, to represent object attributes in an allocentric manner which does not depend on viewpoint. Moreover, it allows us to disentangle object dynamics and summarize their trajectories as time-abstracted, view-invariant, per-object properties. We demonstrate these capabilities, as well as the model's performance in terms of view synthesis and instance segmentation, across three procedurally generated video datasets.