Play and Learn: Using Video Games to Train Computer Vision Models
This addresses the data scarcity problem for computer vision researchers by leveraging accessible synthetic data, though it is incremental as it builds on existing domain adaptation techniques.
The paper tackled the problem of whether synthetic RGB images from video games can improve computer vision models on real-world tasks, finding that networks trained on synthetic data achieve similar test error to those trained on real data for dense image classification and can provide similar or better results with domain adaptation.
Video games are a compelling source of annotated data as they can readily provide fine-grained groundtruth for diverse tasks. However, it is not clear whether the synthetically generated data has enough resemblance to the real-world images to improve the performance of computer vision models in practice. We present experiments assessing the effectiveness on real-world data of systems trained on synthetic RGB images that are extracted from a video game. We collected over 60000 synthetic samples from a modern video game with similar conditions to the real-world CamVid and Cityscapes datasets. We provide several experiments to demonstrate that the synthetically generated RGB images can be used to improve the performance of deep neural networks on both image segmentation and depth estimation. These results show that a convolutional network trained on synthetic data achieves a similar test error to a network that is trained on real-world data for dense image classification. Furthermore, the synthetically generated RGB images can provide similar or better results compared to the real-world datasets if a simple domain adaptation technique is applied. Our results suggest that collaboration with game developers for an accessible interface to gather data is potentially a fruitful direction for future work in computer vision.