Deep Diffusion Image Prior for Efficient OOD Adaptation in 3D Inverse Problems
This work addresses the challenge of efficient out-of-distribution adaptation for 3D reconstruction tasks, enabling the use of diffusion priors even when training with ideal data is not feasible.
The paper tackles the problem of adapting generative diffusion priors for 3D inverse problems when there is a discrepancy between training and testing distributions, proposing DDIP and D3IP methods that accelerate adaptation by orders of magnitude and achieve superior performance.
Recent inverse problem solvers that leverage generative diffusion priors have garnered significant attention due to their exceptional quality. However, adaptation of the prior is necessary when there exists a discrepancy between the training and testing distributions. In this work, we propose deep diffusion image prior (DDIP), which generalizes the recent adaptation method of SCD by introducing a formal connection to the deep image prior. Under this framework, we propose an efficient adaptation method dubbed D3IP, specified for 3D measurements, which accelerates DDIP by orders of magnitude while achieving superior performance. D3IP enables seamless integration of 3D inverse solvers and thus leads to coherent 3D reconstruction. Moreover, we show that meta-learning techniques can also be applied to yield even better performance. We show that our method is capable of solving diverse 3D reconstructive tasks from the generative prior trained only with phantom images that are vastly different from the training set, opening up new opportunities of applying diffusion inverse solvers even when training with gold standard data is impossible. Code: https://github.com/HJ-harry/DDIP3D