GCISG: Guided Causal Invariant Learning for Improved Syn-to-real Generalization
This addresses the challenge of data scarcity in machine learning by improving model generalization from synthetic to real data, though it is incremental as it builds on existing causal frameworks.
The paper tackles the problem of poor generalization when training deep learning models on synthetic data due to domain gaps, by proposing a causal invariance learning method that achieves state-of-the-art performance on visual syn-to-real domain generalization tasks like image classification and semantic segmentation.
Training a deep learning model with artificially generated data can be an alternative when training data are scarce, yet it suffers from poor generalization performance due to a large domain gap. In this paper, we characterize the domain gap by using a causal framework for data generation. We assume that the real and synthetic data have common content variables but different style variables. Thus, a model trained on synthetic dataset might have poor generalization as the model learns the nuisance style variables. To that end, we propose causal invariance learning which encourages the model to learn a style-invariant representation that enhances the syn-to-real generalization. Furthermore, we propose a simple yet effective feature distillation method that prevents catastrophic forgetting of semantic knowledge of the real domain. In sum, we refer to our method as Guided Causal Invariant Syn-to-real Generalization that effectively improves the performance of syn-to-real generalization. We empirically verify the validity of proposed methods, and especially, our method achieves state-of-the-art on visual syn-to-real domain generalization tasks such as image classification and semantic segmentation.