Photo-realistic Neural Domain Randomization
This work addresses the problem of domain adaptation for computer vision tasks like object detection and depth estimation, offering a scalable solution for researchers and practitioners using synthetic data, though it is incremental as it builds on existing neural rendering and domain randomization approaches.
The paper tackles the sim-to-real domain gap in synthetic data by introducing Photo-realistic Neural Domain Randomization (PNDR), a method that uses neural rendering to generate high-quality, randomized images from scene geometry, and demonstrates its effectiveness by significantly outperforming state-of-the-art methods in 6D object detection and monocular depth estimation tasks.
Synthetic data is a scalable alternative to manual supervision, but it requires overcoming the sim-to-real domain gap. This discrepancy between virtual and real worlds is addressed by two seemingly opposed approaches: improving the realism of simulation or foregoing realism entirely via domain randomization. In this paper, we show that the recent progress in neural rendering enables a new unified approach we call Photo-realistic Neural Domain Randomization (PNDR). We propose to learn a composition of neural networks that acts as a physics-based ray tracer generating high-quality renderings from scene geometry alone. Our approach is modular, composed of different neural networks for materials, lighting, and rendering, thus enabling randomization of different key image generation components in a differentiable pipeline. Once trained, our method can be combined with other methods and used to generate photo-realistic image augmentations online and significantly more efficiently than via traditional ray-tracing. We demonstrate the usefulness of PNDR through two downstream tasks: 6D object detection and monocular depth estimation. Our experiments show that training with PNDR enables generalization to novel scenes and significantly outperforms the state of the art in terms of real-world transfer.