Neural-Sim: Learning to Generate Training Data with NeRF
This addresses the time-consuming and domain-mapping challenges in data collection for computer vision practitioners, though it is incremental as it builds on existing synthetic data and NeRF methods.
The paper tackles the problem of generating synthetic training data for computer vision by introducing a fully differentiable pipeline using Neural Radiance Fields (NeRFs) in a closed-loop with a target loss function, enabling on-demand data generation without human labor to maximize task accuracy, as demonstrated on object detection tasks with a new 'YCB-in-the-Wild' dataset.
Training computer vision models usually requires collecting and labeling vast amounts of imagery under a diverse set of scene configurations and properties. This process is incredibly time-consuming, and it is challenging to ensure that the captured data distribution maps well to the target domain of an application scenario. Recently, synthetic data has emerged as a way to address both of these issues. However, existing approaches either require human experts to manually tune each scene property or use automatic methods that provide little to no control; this requires rendering large amounts of random data variations, which is slow and is often suboptimal for the target domain. We present the first fully differentiable synthetic data pipeline that uses Neural Radiance Fields (NeRFs) in a closed-loop with a target application's loss function. Our approach generates data on-demand, with no human labor, to maximize accuracy for a target task. We illustrate the effectiveness of our method on synthetic and real-world object detection tasks. We also introduce a new "YCB-in-the-Wild" dataset and benchmark that provides a test scenario for object detection with varied poses in real-world environments.