Domain Generalization Guided by Gradient Signal to Noise Ratio of Parameters
This work addresses domain shift issues in machine learning, offering an incremental improvement over existing regularization techniques for domain generalization.
The paper tackles overfitting to source domains in deep neural networks by proposing a gradient-signal-to-noise ratio (GSNR)-based dropout method, achieving competitive results on domain generalization benchmarks for classification and face anti-spoofing.
Overfitting to the source domain is a common issue in gradient-based training of deep neural networks. To compensate for the over-parameterized models, numerous regularization techniques have been introduced such as those based on dropout. While these methods achieve significant improvements on classical benchmarks such as ImageNet, their performance diminishes with the introduction of domain shift in the test set i.e. when the unseen data comes from a significantly different distribution. In this paper, we move away from the classical approach of Bernoulli sampled dropout mask construction and propose to base the selection on gradient-signal-to-noise ratio (GSNR) of network's parameters. Specifically, at each training step, parameters with high GSNR will be discarded. Furthermore, we alleviate the burden of manually searching for the optimal dropout ratio by leveraging a meta-learning approach. We evaluate our method on standard domain generalization benchmarks and achieve competitive results on classification and face anti-spoofing problems.