JointNet: Extending Text-to-Image Diffusion for Dense Distribution Modeling
This work addresses the need for joint modeling of images and dense modalities in computer vision, offering an incremental improvement by building on existing diffusion models.
The paper tackles the problem of modeling joint distributions of images and dense modalities like depth maps by introducing JointNet, which extends a pre-trained text-to-image diffusion model with a new branch for the dense modality, achieving efficient learning while maintaining generalization across applications such as RGBD generation and depth prediction.
We introduce JointNet, a novel neural network architecture for modeling the joint distribution of images and an additional dense modality (e.g., depth maps). JointNet is extended from a pre-trained text-to-image diffusion model, where a copy of the original network is created for the new dense modality branch and is densely connected with the RGB branch. The RGB branch is locked during network fine-tuning, which enables efficient learning of the new modality distribution while maintaining the strong generalization ability of the large-scale pre-trained diffusion model. We demonstrate the effectiveness of JointNet by using RGBD diffusion as an example and through extensive experiments, showcasing its applicability in a variety of applications, including joint RGBD generation, dense depth prediction, depth-conditioned image generation, and coherent tile-based 3D panorama generation.