GO-Renderer: Generative Object Rendering with 3D-aware Controllable Video Diffusion Models
This addresses the challenge of realistic object rendering for applications in computer graphics and vision, though it is incremental by integrating existing 3D reconstruction and diffusion models.
The paper tackles the problem of rendering 3D objects with accurate viewpoint control and high-quality appearance under arbitrary lighting conditions, achieving state-of-the-art performance in tasks like synthesizing images from new viewpoints and inserting objects into videos.
Reconstructing a renderable 3D model from images is a useful but challenging task. Recent feedforward 3D reconstruction methods have demonstrated remarkable success in efficiently recovering geometry, but still cannot accurately model the complex appearances of these 3D reconstructed models. Recent diffusion-based generative models can synthesize realistic images or videos of an object using reference images without explicitly modeling its appearance, which provides a promising direction for object rendering, but lacks accurate control over the viewpoints. In this paper, we propose GO-Renderer, a unified framework integrating the reconstructed 3D proxies to guide the video generative models to achieve high-quality object rendering on arbitrary viewpoints under arbitrary lighting conditions. Our method not only enjoys the accurate viewpoint control using the reconstructed 3D proxy but also enables high-quality rendering in different lighting environments using diffusion generative models without explicitly modeling complex materials and lighting. Extensive experiments demonstrate that GO-Renderer achieves state-of-the-art performance across the object rendering tasks, including synthesizing images on new viewpoints, rendering the objects in a novel lighting environment, and inserting an object into an existing video.