Unsupervised Keypoints from Pretrained Diffusion Models
This work addresses the challenge of robust unsupervised keypoint detection for computer vision applications, offering a novel approach that can enhance performance in scenarios with limited labeled data.
The paper tackles the problem of unsupervised keypoint learning by leveraging text-to-image diffusion models to find text embeddings that cause consistent attention to compact image regions, achieving significantly improved accuracy on multiple datasets, sometimes outperforming supervised methods, especially for non-aligned and less curated data.
Unsupervised learning of keypoints and landmarks has seen significant progress with the help of modern neural network architectures, but performance is yet to match the supervised counterpart, making their practicability questionable. We leverage the emergent knowledge within text-to-image diffusion models, towards more robust unsupervised keypoints. Our core idea is to find text embeddings that would cause the generative model to consistently attend to compact regions in images (i.e. keypoints). To do so, we simply optimize the text embedding such that the cross-attention maps within the denoising network are localized as Gaussians with small standard deviations. We validate our performance on multiple datasets: the CelebA, CUB-200-2011, Tai-Chi-HD, DeepFashion, and Human3.6m datasets. We achieve significantly improved accuracy, sometimes even outperforming supervised ones, particularly for data that is non-aligned and less curated. Our code is publicly available and can be found through our project page: https://ubc-vision.github.io/StableKeypoints/