Learning predictable and robust neural representations by straightening image sequences
This work addresses the challenge of creating robust and interpretable neural representations for unsupervised learning in computer vision, offering a biologically-inspired method that enhances prediction and resilience, though it is incremental in building on existing self-supervised learning frameworks.
The authors tackled the problem of learning neural representations that are both predictable and robust by introducing a self-supervised objective based on straightening image sequences, inspired by biological vision. They demonstrated that this approach leads to embeddings that factorize object attributes and are more robust to noise and adversarial attacks compared to prior methods, with improvements transferable as a regularizer.
Prediction is a fundamental capability of all living organisms, and has been proposed as an objective for learning sensory representations. Recent work demonstrates that in primate visual systems, prediction is facilitated by neural representations that follow straighter temporal trajectories than their initial photoreceptor encoding, which allows for prediction by linear extrapolation. Inspired by these experimental findings, we develop a self-supervised learning (SSL) objective that explicitly quantifies and promotes straightening. We demonstrate the power of this objective in training deep feedforward neural networks on smoothly-rendered synthetic image sequences that mimic commonly-occurring properties of natural videos. The learned model contains neural embeddings that are predictive, but also factorize the geometric, photometric, and semantic attributes of objects. The representations also prove more robust to noise and adversarial attacks compared to previous SSL methods that optimize for invariance to random augmentations. Moreover, these beneficial properties can be transferred to other training procedures by using the straightening objective as a regularizer, suggesting a broader utility for straightening as a principle for robust unsupervised learning.