Toward a Diffusion-Based Generalist for Dense Vision Tasks
This work addresses the need for versatile vision models that can handle various tasks simultaneously, though it is incremental as it builds on existing diffusion-based approaches.
The paper tackles the problem of building a generalist model for multiple dense vision tasks by unifying them as conditional image generation using diffusion models, and achieves competitive performance across four different tasks.
Building generalized models that can solve many computer vision tasks simultaneously is an intriguing direction. Recent works have shown image itself can be used as a natural interface for general-purpose visual perception and demonstrated inspiring results. In this paper, we explore diffusion-based vision generalists, where we unify different types of dense prediction tasks as conditional image generation and re-purpose pre-trained diffusion models for it. However, directly applying off-the-shelf latent diffusion models leads to a quantization issue. Thus, we propose to perform diffusion in pixel space and provide a recipe for finetuning pre-trained text-to-image diffusion models for dense vision tasks. In experiments, we evaluate our method on four different types of tasks and show competitive performance to the other vision generalists.