Exploiting Diffusion Prior for Generalizable Dense Prediction
This work addresses the problem of generalizable dense prediction for arbitrary images, offering a novel method that improves accuracy despite limited training data.
The paper tackles the domain gap between text-to-image diffusion models and dense prediction tasks by introducing DMP, a pipeline that uses pre-trained diffusion models as a prior, achieving state-of-the-art performance across five tasks like 3D property estimation and semantic segmentation.
Contents generated by recent advanced Text-to-Image (T2I) diffusion models are sometimes too imaginative for existing off-the-shelf dense predictors to estimate due to the immitigable domain gap. We introduce DMP, a pipeline utilizing pre-trained T2I models as a prior for dense prediction tasks. To address the misalignment between deterministic prediction tasks and stochastic T2I models, we reformulate the diffusion process through a sequence of interpolations, establishing a deterministic mapping between input RGB images and output prediction distributions. To preserve generalizability, we use low-rank adaptation to fine-tune pre-trained models. Extensive experiments across five tasks, including 3D property estimation, semantic segmentation, and intrinsic image decomposition, showcase the efficacy of the proposed method. Despite limited-domain training data, the approach yields faithful estimations for arbitrary images, surpassing existing state-of-the-art algorithms.