Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer
This work addresses a fundamental bottleneck in computer vision and graphics by making rendering pipelines accessible to ML, which could improve generalization in tasks like 3D reconstruction and generation.
The paper tackles the problem of enabling gradient-based machine learning models to understand 3D image formation by introducing DIB-R, a differentiable rendering framework that allows gradients to be computed for all pixels, and demonstrates its application in single-image 3D object prediction and 3D textured object generation using only 2D supervision.
Many machine learning models operate on images, but ignore the fact that images are 2D projections formed by 3D geometry interacting with light, in a process called rendering. Enabling ML models to understand image formation might be key for generalization. However, due to an essential rasterization step involving discrete assignment operations, rendering pipelines are non-differentiable and thus largely inaccessible to gradient-based ML techniques. In this paper, we present {\emph DIB-R}, a differentiable rendering framework which allows gradients to be analytically computed for all pixels in an image. Key to our approach is to view foreground rasterization as a weighted interpolation of local properties and background rasterization as a distance-based aggregation of global geometry. Our approach allows for accurate optimization over vertex positions, colors, normals, light directions and texture coordinates through a variety of lighting models. We showcase our approach in two ML applications: single-image 3D object prediction, and 3D textured object generation, both trained using exclusively using 2D supervision. Our project website is: https://nv-tlabs.github.io/DIB-R/