Learning Image-Adaptive Codebooks for Class-Agnostic Image Restoration
This addresses the limitation of existing discrete generative priors that require separate training for different image categories, enabling restoration of arbitrary natural images.
The paper tackles the problem of class-agnostic image restoration by proposing AdaCode, which learns image-adaptive codebooks instead of category-specific ones, achieving state-of-the-art performance on tasks like super-resolution and inpainting.
Recent work on discrete generative priors, in the form of codebooks, has shown exciting performance for image reconstruction and restoration, as the discrete prior space spanned by the codebooks increases the robustness against diverse image degradations. Nevertheless, these methods require separate training of codebooks for different image categories, which limits their use to specific image categories only (e.g. face, architecture, etc.), and fail to handle arbitrary natural images. In this paper, we propose AdaCode for learning image-adaptive codebooks for class-agnostic image restoration. Instead of learning a single codebook for each image category, we learn a set of basis codebooks. For a given input image, AdaCode learns a weight map with which we compute a weighted combination of these basis codebooks for adaptive image restoration. Intuitively, AdaCode is a more flexible and expressive discrete generative prior than previous work. Experimental results demonstrate that AdaCode achieves state-of-the-art performance on image reconstruction and restoration tasks, including image super-resolution and inpainting.