Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding
This work addresses the challenge of applying disentanglement methods to natural data, which is crucial for advancing unsupervised learning in real-world AI applications, though it builds incrementally on prior temporal assumptions.
The paper tackled the problem of unsupervised nonlinear disentanglement of factors in natural videos by introducing SlowVAE, which uses a sparse prior on temporal transitions to learn disentangled representations without assuming constant factors, achieving state-of-the-art results on benchmark datasets and demonstrating transferability to natural video datasets.
We construct an unsupervised learning model that achieves nonlinear disentanglement of underlying factors of variation in naturalistic videos. Previous work suggests that representations can be disentangled if all but a few factors in the environment stay constant at any point in time. As a result, algorithms proposed for this problem have only been tested on carefully constructed datasets with this exact property, leaving it unclear whether they will transfer to natural scenes. Here we provide evidence that objects in segmented natural movies undergo transitions that are typically small in magnitude with occasional large jumps, which is characteristic of a temporally sparse distribution. We leverage this finding and present SlowVAE, a model for unsupervised representation learning that uses a sparse prior on temporally adjacent observations to disentangle generative factors without any assumptions on the number of changing factors. We provide a proof of identifiability and show that the model reliably learns disentangled representations on several established benchmark datasets, often surpassing the current state-of-the-art. We additionally demonstrate transferability towards video datasets with natural dynamics, Natural Sprites and KITTI Masks, which we contribute as benchmarks for guiding disentanglement research towards more natural data domains.