Automated Synthetic-to-Real Generalization
This addresses the laborious and heuristic tuning needed for synthetic-to-real generalization in computer vision, though it is incremental as it builds on existing practices like ImageNet pre-training.
The paper tackles the problem of models trained on synthetic images having degraded generalization to real data by proposing a learning-to-optimize strategy to automate layer-wise learning rate selection, which significantly improves synthetic-to-real generalization without using real data and benefits downstream tasks like domain adaptation.
Models trained on synthetic images often face degraded generalization to real data. As a convention, these models are often initialized with ImageNet pre-trained representation. Yet the role of ImageNet knowledge is seldom discussed despite common practices that leverage this knowledge to maintain the generalization ability. An example is the careful hand-tuning of early stopping and layer-wise learning rates, which is shown to improve synthetic-to-real generalization but is also laborious and heuristic. In this work, we explicitly encourage the synthetically trained model to maintain similar representations with the ImageNet pre-trained model, and propose a \textit{learning-to-optimize (L2O)} strategy to automate the selection of layer-wise learning rates. We demonstrate that the proposed framework can significantly improve the synthetic-to-real generalization performance without seeing and training on real data, while also benefiting downstream tasks such as domain adaptation. Code is available at: https://github.com/NVlabs/ASG.