Diffusion Models as Masked Autoencoders
This work addresses the challenge of generative pre-training for visual data, offering a novel approach that enhances representation learning for computer vision applications, though it builds incrementally on existing diffusion and autoencoder methods.
The paper tackles the problem of improving visual representation learning by proposing DiffMAE, which conditions diffusion models on masked input to function as masked autoencoders, resulting in strong initialization for downstream tasks, high-quality image inpainting, and state-of-the-art video classification accuracy.
There has been a longstanding belief that generation can facilitate a true understanding of visual data. In line with this, we revisit generatively pre-training visual representations in light of recent interest in denoising diffusion models. While directly pre-training with diffusion models does not produce strong representations, we condition diffusion models on masked input and formulate diffusion models as masked autoencoders (DiffMAE). Our approach is capable of (i) serving as a strong initialization for downstream recognition tasks, (ii) conducting high-quality image inpainting, and (iii) being effortlessly extended to video where it produces state-of-the-art classification accuracy. We further perform a comprehensive study on the pros and cons of design choices and build connections between diffusion models and masked autoencoders.