Cut-and-Splat: Leveraging Gaussian Splatting for Synthetic Data Generation
This addresses the need for cheap, high-quality labeled data for computer vision, though it is incremental as it builds on Gaussian Splatting for a specific application.
The paper tackles the problem of generating realistic synthetic images for training computer vision models by proposing a pipeline that uses Gaussian Splatting to render objects from videos onto random backgrounds, achieving superior performance over existing methods like Cut-and-Paste and Diffusion models.
Generating synthetic images is a useful method for cheaply obtaining labeled data for training computer vision models. However, obtaining accurate 3D models of relevant objects is necessary, and the resulting images often have a gap in realism due to challenges in simulating lighting effects and camera artifacts. We propose using the novel view synthesis method called Gaussian Splatting to address these challenges. We have developed a synthetic data pipeline for generating high-quality context-aware instance segmentation training data for specific objects. This process is fully automated, requiring only a video of the target object. We train a Gaussian Splatting model of the target object and automatically extract the object from the video. Leveraging Gaussian Splatting, we then render the object on a random background image, and monocular depth estimation is employed to place the object in a believable pose. We introduce a novel dataset to validate our approach and show superior performance over other data generation approaches, such as Cut-and-Paste and Diffusion model-based generation.