On minimal variations for unsupervised representation learning
This work addresses the foundational problem of improving unsupervised learning guidelines for researchers and practitioners, but it appears incremental as it refines existing assumptions rather than introducing a new method.
The paper identifies that unsupervised representation learning methods assume target functions for downstream tasks have low variations in densely populated regions, and proposes minimal variations as a guiding principle to improve self-supervised learning algorithms.
Unsupervised representation learning aims at describing raw data efficiently to solve various downstream tasks. It has been approached with many techniques, such as manifold learning, diffusion maps, or more recently self-supervised learning. Those techniques are arguably all based on the underlying assumption that target functions, associated with future downstream tasks, have low variations in densely populated regions of the input space. Unveiling minimal variations as a guiding principle behind unsupervised representation learning paves the way to better practical guidelines for self-supervised learning algorithms.