SODA: Bottleneck Diffusion Models for Representation Learning
This addresses the challenge of learning rich visual representations without supervision, showing diffusion models can be effective for both generation and representation learning, though it builds incrementally on existing diffusion model concepts.
The authors tackled the problem of using diffusion models for representation learning by introducing SODA, a self-supervised diffusion model with a bottleneck between encoder and decoder, achieving ImageNet linear-probe classification and performing reconstruction, editing, and synthesis tasks across datasets.
We introduce SODA, a self-supervised diffusion model, designed for representation learning. The model incorporates an image encoder, which distills a source view into a compact representation, that, in turn, guides the generation of related novel views. We show that by imposing a tight bottleneck between the encoder and a denoising decoder, and leveraging novel view synthesis as a self-supervised objective, we can turn diffusion models into strong representation learners, capable of capturing visual semantics in an unsupervised manner. To the best of our knowledge, SODA is the first diffusion model to succeed at ImageNet linear-probe classification, and, at the same time, it accomplishes reconstruction, editing and synthesis tasks across a wide range of datasets. Further investigation reveals the disentangled nature of its emergent latent space, that serves as an effective interface to control and manipulate the model's produced images. All in all, we aim to shed light on the exciting and promising potential of diffusion models, not only for image generation, but also for learning rich and robust representations.